Image processing apparatus and method

ABSTRACT

An image processing apparatus and a method for processing an image. The image processing apparatus includes circuitry configured to set a current quantization parameter for a current coding block in a layer lower than a layer of a largest coding block and a quantization parameter based on a difference between the current quantization parameter for the current coding block and a slice quantization parameter for a slice in which the current coding block is located. The circuitry quantizes image data to generate quantized data based on the current quantization parameter. Further, the circuitry codes the generated quantized data in a coding block including a hierarchical structure to generate a code data including the quantization parameter.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. Ser. No. 13/700,156 filedNov. 27, 2012, the entire content of which is incorporated herein byreference. U.S. Ser. No. 13/700,156 is a national stage ofPCT/JP11/062797 filed Jun. 3, 2011, which claims priority under 35U.S.C. 119 to Japanese Application Nos. 2010-129414 filed Jun. 4, 2010;2010-222300 filed Sep. 30, 2010; 2011-053479 filed Mar. 10, 2011; and2011-054816 filed Mar. 11, 2011.

TECHNICAL FIELD

The present technology relates to an image processing apparatus and amethod and relates to an image processing apparatus and a method forperforming a quantization process or an inverse quantization process.

BACKGROUND ART

Recently, an apparatus complying with a system such as MPEG (MovingPicture Experts Group), which digitally treats image information andcompresses the same by an orthogonal transform such as a discrete cosinetransform and motion compensation by using redundancy specific to theimage information in order to efficiently transmit and accumulate theinformation at that time, has been widely used in both of informationdistribution in a broadcast station and information reception instandard home.

These days, there are growing needs for higher compression coding suchas coding of an image of approximately 4096×2048 pixels, four times asmany as those for a high-definition image, or distribution of thehigh-definition image in an environment of limited transmission capacitysuch as the Internet. Therefore, VCEG under ITU-T continuously studiesimprovement in coding efficiency.

A pixel size of a macroblock being a partial area of the image, which isa division unit (unit of coding process) of the image at the time ofimage coding in MPEG1, MPEG2, and ITU-T H.264/MPEG4-AVC beingconventional image coding systems, is always 16×16 pixels. On the otherhand, Non-Patent Document 1 suggests extending the number of pixels ofthe macroblock horizontally and vertically as elemental technology ofnext-generation image coding standard. This also suggests using themacroblock of 32×32 pixels and 64×64 pixels in addition to the pixelsize of the macroblock of 16×16 pixels defined by the MPEG1, MPEG2,ITU-T H.264/MPEG4-AVC and the like. This is for improving the codingefficiency by performing the motion compensation and the orthogonaltransform in a unit of larger area in areas with similar motion becauseit is predicted that the pixel size of the image to be coded increaseshorizontally and vertically in the future such as UHD (Ultra HighDefinition; 4000 pixels×2000 pixels), for example.

Non-Patent Document 1 adopts a hierarchical structure, thereby defininga larger block as a superset for the block not larger than 16×16 pixelswhile maintaining compatibility with the macroblock of current AVC.

Although Non-Patent Document 1 suggests applying the extended macroblockto inter slice, Non-Patent Document 2 suggests applying the extendedmacroblock to intra slice.

CITATION LIST Non-Patent Documents

-   Non-Patent Document 1: Peisong Chenn, Yan Ye, Marta Karczewicz,    “Video Coding Using Extended Block Sizes”, COM16-C123-E, Qualcomm    Inc-   Non-Patent Document 2: Sung-Chang Lim, Hahyun Lee, Jinho Lee, Jongho    Kim, Haechul Choi, Seyoon Jeong, Jin Soo Choi, “Intra coding using    extended block size”, VCEG-AL28, July, 2009

SUMMARY OF THE INVENTION Problems to be Solved by the Invention

When the extended macroblock as suggested in Non-Patent Documents 1 and2 is applied, possibility that a flat area and an area including textureare mixed in a single macroblock becomes high.

However, since only one quantization parameter may be specified for onemacroblock in Non-Patent Documents 1 and 2, it is difficult to performadaptive quantization according to characteristics of the respectiveareas in a plane.

The present technology has been realized in view of such circumstancesand an object thereof is to inhibit subjective image quality of adecoded image from deteriorating by performing a more appropriatequantization process.

Solution to Problems

An aspect of the present technology is an image processing apparatus,including: a decoding unit which decodes a coded stream to generatequantized data; a setting unit which sets a quantization parameter usedwhen the quantized data generated by the decoding unit is inverselyquantized for a coding unit in a layer lower than a reference codingunit in a reference layer of the coding unit being a unit of codingprocess when image data is coded; and an inverse quantization unit whichinversely quantizes the quantized data generated by the decoding unit byusing the quantization parameter set by the setting unit.

The setting unit may set the quantization parameter for a current codingunit by using a difference quantization parameter indicating adifference value between the quantization parameter set for the currentcoding unit being a target of an inverse quantization process and thequantization parameter set for the coding unit in the same layer as thecurrent coding unit.

The difference quantization parameter may be the difference valuebetween the quantization parameter set for the current coding unit andthe quantization parameter set for the coding unit decoded before thecurrent coding unit in order of decoding process.

The difference quantization parameter may be the difference valuebetween the quantization parameter set for the current coding unit andthe quantization parameter set for the coding unit decoded immediatelybefore the current coding unit in the order of decoding process.

The reference coding unit may be a largest coding unit being the codingunit in a highest layer.

The image processing apparatus may further include: a receiving unitwhich receives the coded stream and minimum coding unit size dataindicating a minimum size of the coding unit for which the differencequantization parameter is set, and the setting unit may set thequantization parameter for the current coding unit according to theminimum coding unit size data received by the receiving unit.

The receiving unit may obtain the minimum coding unit size data from aslice header of the coded stream.

When a size indicated by the minimum coding unit size data is 16 pixels,the difference quantization parameter for the coding unit of which sizeis smaller than 16 pixels may be set to 0.

The setting unit may set the quantization parameter for a current codingunit by using a difference quantization parameter indicating adifference value between the quantization parameter set for the currentcoding unit being a target of a decoding process and the quantizationparameter set for a slice to which the current coding unit belongs.

The setting unit may set the quantization parameter for the currentcoding unit by using the difference quantization parameter indicatingthe difference value between the quantization parameter set for thecurrent coding unit and the quantization parameter set for the slice towhich the current coding unit belongs when the current coding unit is afirst coding unit in order of decoding process in a layer of thereference coding unit.

The reference coding unit may be a largest coding unit being the codingunit in a highest layer.

The image processing apparatus may further include: a receiving unitwhich receives the coded stream and minimum coding unit size dataindicating a minimum size of the coding unit for which the differencequantization parameter is set, and the setting unit may set thequantization parameter for the current coding unit according to theminimum coding unit size data received by the receiving unit.

The receiving unit may obtain the minimum coding unit size data from aslice header of the coded stream.

The difference quantization parameter for the coding unit of which sizeis smaller than 16 pixels may be set to 0 when a size indicated by theminimum coding unit size data is 16 pixels.

The setting unit may set the quantization parameter set for thereference coding unit as the quantization parameter set for the codingunit in a layer lower than the reference coding unit when a value of thedifference quantization parameter is 0 for the coding unit in the layerlower than the reference coding unit.

The image processing apparatus may further include: a receiving unitwhich receives difference identification data for identifying whetherthe value of the difference quantization parameter is 0 for the codingunit in the layer lower than the reference coding unit, and the settingunit may set the quantization parameter set for the reference codingunit as the quantization parameter set for the coding unit in the layerlower than the reference coding unit by using the differenceidentification data received by the receiving unit.

An aspect of the present technology is an image processing method,including: generating quantized data by decoding a coded stream; settinga quantization parameter used when the generated quantized data isinversely quantized for a coding unit in a layer lower than a referencecoding unit in a reference layer of the coding unit being a unit ofcoding process when image data is coded; and inversely quantizing thegenerated quantized data by using the set quantization parameter.

Another aspect of the present technology is an image processingapparatus, including: a setting unit which sets a quantization parameterused when image data is quantized for a coding unit in a layer lowerthan a reference coding unit in a reference layer of the coding unitbeing a unit of coding process when the image data is coded; aquantization unit which generates quantized data by quantizing the imagedata by using the quantization parameter set by the setting unit; and acoding unit which codes the quantized data generated by the quantizationunit to generate a coded stream.

The setting unit may set a difference quantization parameter indicatinga difference value between the quantization parameter set for a currentcoding unit being a target of a coding process and the quantizationparameter set for the coding unit in the same layer as the currentcoding unit, and the image processing apparatus may further include: atransmitting unit which transmits the difference quantization parameterset by the setting unit and the coded stream generated by the codingunit.

The setting unit may set, as the difference quantization parameter, thedifference value between the quantization parameter set for the currentcoding unit and the quantization parameter set for the coding unit codedbefore the current coding unit in order of coding process.

The setting unit may set, as the difference quantization parameter, thedifference value between the quantization parameter set for the currentcoding unit and the quantization parameter set for the coding unit codedimmediately before the current coding unit in the order of codingprocess.

The reference coding unit may be a largest coding unit being the codingunit in a highest layer.

The setting unit may set minimum coding unit size data indicating aminimum size of the coding unit for which the difference quantizationparameter is set, and the transmitting unit may transmit the minimumcoding unit size data set by the setting unit.

The transmitting unit may add, as a slice header, the minimum codingunit size data set by the setting unit to syntax of the coded streamgenerated by the coding unit.

The setting unit may set the difference quantization parameter for thecoding unit of which size is smaller than 16 pixels to 0 when a sizeindicated by the minimum coding unit size data is set to 16 pixels.

The setting unit may set a difference quantization parameter indicatinga difference value between the quantization parameter set for a currentcoding unit being a target of a coding process and the quantizationparameter set for a slice to which the current coding unit belongs, andthe image processing apparatus may further include: a transmitting unitwhich transmits the difference quantization parameter set by the settingunit and the coded stream generated by the coding unit.

The setting unit may set, as the difference quantization parameter, thedifference value between the quantization parameter set for the currentcoding unit and the quantization parameter set for the slice to whichthe current coding unit belongs when the current coding unit is a firstcoding unit in order of coding process in a layer of the referencecoding unit.

The reference coding unit may be a largest coding unit being the codingunit in a highest layer.

The setting unit may set minimum coding unit size data indicating aminimum size of the coding unit for which the difference quantizationparameter is set, and the transmitting unit may transmit the minimumcoding unit size data set by the setting unit.

The transmitting unit may add, as a slice header, the minimum codingunit size data set by the setting unit to syntax of the coded streamgenerated by the coding unit.

The setting unit may set the difference quantization parameter for thecoding unit of which size is smaller than 16 pixels to 0 when a sizeindicated by the minimum coding unit size data is set to 16 pixels.

The setting unit may set the quantization parameter set for thereference coding unit as the quantization parameter set for the codingunit in the layer lower than the reference coding unit when a value ofthe difference quantization parameter is set to 0 for the coding unit inthe layer lower than the reference coding unit.

The setting unit may set difference identification data for identifyingwhether the value of the difference quantization parameter is 0 for thecoding unit in the layer lower than the reference coding unit, and theimage processing apparatus may further include: a transmitting unitwhich transmits the difference identification data set by the settingunit and the coded stream generated by the coding unit.

Another aspect of the present technology is an image processing method,including: setting a quantization parameter used when image data isquantized for a coding unit in a layer lower than a reference codingunit in a reference layer of the coding unit being a unit of codingprocess when the image data is coded; generating quantized data byquantizing the image data by using the set quantization parameter; andgenerating a coded stream by coding the generated quantized data.

In an aspect of the present technology, the coded stream is decoded andthe quantized data is generated, the quantization parameter used whenthe generated quantized data is inversely quantized is set for thecoding unit in the layer lower than the reference coding unit in thereference layer of the coding unit being the unit of coding process whenthe image data is coded, and the generated quantized data is inverselyquantized by using the set quantization parameter.

In another aspect of the present technology, the quantization parameterused when the image data is quantized is set for the coding unit in thelayer lower than the reference coding unit in the reference layer of thecoding unit being the unit of coding process when the image data iscoded, the image data is quantized and the quantized data is generatedby using the set quantization parameter, and the generated quantizeddata is coded and the coded stream is generated.

Effects of the Invention

According to the present technology, the quantization process or theinverse quantization process may be performed more appropriately.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram illustrating a principal configuration exampleof an image coding apparatus to which the present technology is applied.

FIG. 2 is a view illustrating an example of correspondence relationshipbetween a quantization parameter for a luminance signal and aquantization parameter for a chrominance signal.

FIG. 3 is a view illustrating an example of a macroblock.

FIG. 4 is a view illustrating another example of the macroblock.

FIG. 5 is a block diagram illustrating a detailed configuration exampleof a quantization unit.

FIG. 6 is a view illustrating an example of an image in a unit ofmacroblock.

FIG. 7 is a flowchart illustrating an example of a flow of a codingprocess.

FIG. 8 is a flowchart illustrating an example of a flow of aquantization parameter calculation process.

FIG. 9 is a block diagram illustrating a principal configuration exampleof an image decoding apparatus to which the present technology isapplied.

FIG. 10 is a block diagram illustrating a detailed configuration exampleof an inverse quantization unit.

FIG. 11 is a flowchart illustrating an example of a flow of a decodingprocess.

FIG. 12 is a flowchart illustrating an example of a flow of an inversequantization process.

FIG. 13 is a flowchart illustrating another example of the flow of thequantization parameter calculation process.

FIG. 14 is a flowchart illustrating another example of the flow of theinverse quantization process.

FIG. 15 is a view illustrating a configuration example of a coding unit.

FIG. 16 is a view illustrating an example of the quantization parameterassigned to each coding unit.

FIG. 17 is a view illustrating an example of syntax.

FIG. 18 is a block diagram illustrating another configuration example ofthe image coding apparatus to which the present technology is applied.

FIG. 19 is a block diagram illustrating a detailed configuration exampleof a coding unit quantization unit and a rate controller.

FIG. 20 is a flowchart illustrating still another example of the flow ofthe quantization parameter calculation process.

FIG. 21 is a block diagram illustrating another configuration example ofthe image decoding apparatus to which the present technology is applied.

FIG. 22 is a block diagram illustrating a detailed configuration exampleof a coding unit inverse quantization unit.

FIG. 23 is a flowchart illustrating still another example of the flow ofthe inverse quantization process.

FIG. 24 is a view comparing characteristics of methods of calculating aquantization parameter dQP.

FIG. 25 is a view illustrating an example of the quantization parameterassigned to each coding unit.

FIG. 26 is a view illustrating an example of the syntax of a sliceheader.

FIG. 27 is a view illustrating an example of a method of calculatingactivity.

FIG. 28 is a view illustrating a relationship between the quantizationparameter and a quantization scale.

FIG. 29 is a block diagram illustrating still another configurationexample of the image coding apparatus to which the present technology isapplied.

FIG. 30 is a block diagram illustrating a detailed configuration exampleof the coding unit quantization unit, the quantization unit, and therate controller.

FIG. 31 is a flowchart illustrating another example of the flow of thecoding process.

FIG. 32 is a flowchart illustrating an example of a flow of aquantization process.

FIG. 33 is a view illustrating an example of a multi-view image codingsystem.

FIG. 34 is a view illustrating a principal configuration example of themulti-view image coding apparatus to which the present technology isapplied.

FIG. 35 is a view illustrating a principal configuration example of amulti-view image decoding apparatus to which the present technology isapplied.

FIG. 36 is a view illustrating an example of a hierarchical image codingsystem.

FIG. 37 is a view illustrating a principal configuration example of ahierarchical image coding apparatus to which the present technology isapplied.

FIG. 38 is a view illustrating a principal configuration example of ahierarchical image decoding apparatus to which the present technology isapplied.

FIG. 39 is a block diagram illustrating a principal configurationexample of a computer to which the present technology is applied.

FIG. 40 is a block diagram illustrating a principal configurationexample of a television apparatus to which the present technology isapplied.

FIG. 41 is a block diagram illustrating a principal configurationexample of a mobile terminal to which the present technology is applied.

FIG. 42 is a block diagram illustrating a principal configurationexample of a recording/reproducing apparatus to which the presenttechnology is applied.

FIG. 43 is a block diagram illustrating a principal configurationexample of an imaging apparatus to which the present technology isapplied.

MODE FOR CARRYING OUT THE INVENTION

Modes for carrying out the present technology (hereinafter, referred toas embodiments) are hereinafter described. Note that the description isgiven in the following order.

-   -   1. First Embodiment (Image Coding Apparatus)    -   2. Second Embodiment (Image Decoding Apparatus)    -   3. Third Embodiment (Image Coding Apparatus/Image Decoding        Apparatus)    -   4. Fourth Embodiment (Image Coding Apparatus/Image Decoding        Apparatus)    -   5. Fifth Embodiment (Image Coding Apparatus)    -   6. Sixth Embodiment (Multi-View Image Coding/Multi-View Image        Decoding Apparatuses)    -   7. Seventh Embodiment (Hierarchical Image Coding/Hierarchical        Image Decoding Apparatuses)    -   8. Eighth Embodiment (Application)

1. First Embodiment

[Image Coding Apparatus]

FIG. 1 illustrates a configuration of one embodiment of an image codingapparatus as an image processing apparatus to which the presenttechnology is applied.

An image coding apparatus 100 illustrated in FIG. 1 is a codingapparatus, which codes an image in a manner similar to H.264/MPEG(Moving Picture Experts Group)-4 Part 10 (AVC (Advanced Video Coding))(hereinafter, referred to as H.264/AVC) system, for example. However,the image coding apparatus 100 specifies a quantization parameter foreach sub macroblock.

A macroblock is a partial area of the image, which is a unit of processwhen the image is coded. The sub macroblock is a small area obtained bydividing the macroblock into a plurality of parts.

In an example in FIG. 1, the image coding apparatus 100 includes an A/D(Analog/Digital) converter 101, a screen reorder buffer 102, anarithmetic unit 103, an orthogonal transformation unit 104, aquantization unit 105, a lossless coding unit 106, and an accumulationbuffer 107. The image coding apparatus 100 also includes an inversequantization unit 108, an inverse orthogonal transformation unit 109, anarithmetic unit 110, a deblocking filter 111, a frame memory 112, aselector 113, an intra prediction unit 114, a motionprediction/compensation unit 115, a selector 116, and a rate controller117.

The image coding apparatus 100 further includes a sub macroblockquantization unit 121 and a sub macroblock inverse quantization unit122.

The A/D converter 101 A/D converts input image data and outputs the sameto the screen reorder buffer 102 for storage.

The screen reorder buffer 102 reorders the stored image with frames inorder of display into order of frames for coding according to a GOP(Group of Picture) structure. The screen reorder buffer 102 supplies theimage, in which order of the frames has been reordered, to thearithmetic unit 103. The screen reorder buffer 102 supplies the image,in which order of the frames has been reordered, also to the intraprediction unit 114 and the motion prediction/compensation unit 115.

The arithmetic unit 103 subtracts a predicted image supplied from theintra prediction unit 114 or the motion prediction/compensation unit 115through the selector 116 from the image read from the screen reorderbuffer 102 and outputs difference information to the orthogonaltransformation unit 104.

For example, in a case of the image to which intra coding is performed,the arithmetic unit 103 subtracts, from the image read from the screenreorder buffer 102, the predicted image supplied from the intraprediction unit 114. Also, in a case of the image to which inter codingis performed, for example, the arithmetic unit 103 subtracts, from theimage read from the screen reorder buffer 102, the predicted imagesupplied from the motion prediction/compensation unit 115.

The orthogonal transformation unit 104 performs an orthogonal transformsuch as a discrete cosine transform and a Karhunen-Loeve transform tothe difference information supplied from the arithmetic unit 103 andsupplies a transform coefficient to the quantization unit 105.

The quantization unit 105 quantizes the transform coefficient outputfrom the orthogonal transformation unit 104. The quantization unit 105sets the quantization parameter for each sub macroblock, which is anarea smaller than the macroblock, in cooperation with the sub macroblockquantization unit 121 based on information supplied from the ratecontroller 117 and performs quantization. The quantization unit 105supplies the quantized transform coefficient to the lossless coding unit106.

The lossless coding unit 106 performs lossless coding such asvariable-length coding and arithmetic coding to the quantized transformcoefficient.

The lossless coding unit 106 obtains information indicating intraprediction and the like from the intra prediction unit 114 and obtainsinformation indicating an inter prediction mode, motion vectorinformation and the like from the motion prediction/compensation unit115. Meanwhile, the information indicating the intra prediction (intraprediction) is hereinafter also referred to as intra prediction modeinformation. Also, information indicating an information mode indicatinginter prediction (inter prediction) is hereinafter also referred to asinter prediction mode information.

The lossless coding unit 106 codes the quantized transform coefficientand makes various pieces of information such as a filter coefficient,the intra prediction mode information, the inter prediction modeinformation, and the quantization parameter a part of header informationof coded data (multiplexes). The lossless coding unit 106 supplies thecoded data obtained by the coding to the accumulation buffer 107 foraccumulation.

For example, the lossless coding unit 106 performs a lossless codingprocess such as the variable-length coding or the arithmetic coding. Thevariable-length coding includes CAVLC (Context-Adaptive Variable LengthCoding) defined by the H.264/AVC system and the like. The arithmeticcoding includes CABAC (Context-Adaptive Binary Arithmetic Coding) andthe like.

The accumulation buffer 107 temporarily holds the coded data suppliedfrom the lossless coding unit 106 and outputs the same as a coded imagecoded by the H.264/AVC system to subsequent recording apparatus andtransmission channel not illustrated, for example, at a predeterminedtiming.

The transform coefficient quantized by the quantization unit 105 is alsosupplied to the inverse quantization unit 108. The inverse quantizationunit 108 inversely quantizes the quantized transform coefficient by amethod corresponding to the quantization by the quantization unit 105.The inverse quantization unit 108 performs inverse quantization by usingthe quantization parameter for each sub macroblock set by thequantization unit 105 in cooperation with the sub macroblock inversequantization unit 122. The inverse quantization unit 108 supplies anobtained transform coefficient to the inverse orthogonal transformationunit 109.

The inverse orthogonal transformation unit 109 inversely orthogonallytransforms the supplied transform coefficient using a methodcorresponding to an orthogonal transform process by the orthogonaltransformation unit 104. An output obtained by inverse orthogonaltransform (restored difference information) is supplied to thearithmetic unit 110.

The arithmetic unit 110 adds the predicted image supplied from the intraprediction unit 114 or the motion prediction/compensation unit 115through the selector 116 to a result of the inverse orthogonaltransform, that is, the restored difference information supplied fromthe inverse orthogonal transformation unit 109 to obtain a locallydecoded image (decoded image).

For example, when the difference information corresponds to the image towhich the intra coding is performed, the arithmetic unit 110 adds thepredicted image supplied from the intra prediction unit 114 to thedifference information. Also, for example, when the differenceinformation corresponds to the image to which the inter coding isperformed, the arithmetic unit 110 adds the predicted image suppliedfrom the motion prediction/compensation unit 115 to the differenceinformation.

An addition result is supplied to the deblocking filter 111 or the framememory 112.

The deblocking filter 111 removes block distortion of the decoded imageby appropriately performing a deblocking filter process and improvesimage quality by appropriately performing a loop filter process by usinga Wiener filter, for example. The deblocking filter 111 classifies eachpixel and performs an appropriate filter process for each class. Thedeblocking filter 111 supplies a result of the filter process to theframe memory 112.

The frame memory 112 outputs an accumulated reference image to the intraprediction unit 114 or the motion prediction/compensation unit 115through the selector 113 at a predetermined timing.

For example, in the case of the image to which the intra coding isperformed, the frame memory 112 supplies the reference image to theintra prediction unit 114 through the selector 113. Also, for example,when the inter coding is performed, the frame memory 112 supplies thereference image to the motion prediction/compensation unit 115 throughthe selector 113.

When the reference image supplied from the frame memory 112 is the imageto which the intra coding is performed, the selector 113 supplies thereference image to the intra prediction unit 114. Also, when thereference image supplied from the frame memory 112 is the image to whichthe inter coding is performed, the selector 113 supplies the referenceimage to the motion prediction/compensation unit 115.

The intra prediction unit 114 performs the intra prediction (intraprediction) to generate the predicted image by using a pixel value in ascreen. The intra prediction unit 114 performs the intra prediction in aplurality of modes (intra prediction modes).

The intra prediction unit 114 generates the predicted image in all ofthe intra prediction modes and evaluates each predicted image to selectan optimal mode. Upon selecting the optimal intra prediction mode, theintra prediction unit 114 supplies the predicted image generated in theoptimal mode to the arithmetic unit 103 and the arithmetic unit 110through the selector 116.

Also, as described above, the intra prediction unit 114 appropriatelysupplies, to the lossless coding unit 106, the information such as theintra prediction mode information indicating an adopted intra predictionmode.

The motion prediction/compensation unit 115 performs motion predictionby using the input image supplied from the screen reorder buffer 102 andthe reference image supplied from the frame memory 112 through theselector 113, performs a motion compensation process according to adetected motion vector and generates the predicted image (interprediction image information) for the image to which the inter coding isperformed.

The motion prediction/compensation unit 115 performs an inter predictionprocess in all of candidate inter prediction modes to generate thepredicted image. The motion prediction/compensation unit 115 suppliesthe generated predicted image to the arithmetic unit 103 and thearithmetic unit 110 through the selector 116.

Also, the motion prediction/compensation unit 115 supplies the interprediction mode information indicating an adopted inter prediction modeand motion vector information indicating a calculated motion vector tothe lossless coding unit 106.

In the case of the image to which the intra coding is performed, theselector 116 supplies an output of the intra prediction unit 114 to thearithmetic unit 103 and the arithmetic unit 110, and in the case of theimage to which the inter coding is performed, the selector 116 suppliesan output of the motion prediction/compensation unit 115 to thearithmetic unit 103 and the arithmetic unit 110.

The rate controller 117 controls a rate of quantization operation of thequantization unit 105 such that overflow or underflow does not occurbased on a compressed image accumulated in the accumulation buffer 107.The rate controller 117 supplies information indicating complexity ofthe image to the quantization unit 105 for each sub macroblock, which isthe small area obtained by dividing the macroblock into a plurality ofparts.

For example, the rate controller 117 provides activity, which isinformation indicating dispersion of the pixel values, to thequantization unit 105 as the information indicating the complexity ofthe image. It goes without saying that the information indicating thecomplexity of the image may be any information.

The sub macroblock quantization unit 121 obtains the informationindicating the complexity of the image for each sub macroblock from thequantization unit 105, sets a quantization value (quantization step) foreach sub macroblock based on the information, and returns the value tothe quantization unit 105.

The sub macroblock inverse quantization unit 122 obtains thequantization parameter from the inverse quantization unit 108, obtainsthe quantization value for each sub macroblock by using the value of theparameter, and returns the same to the inverse quantization unit 108.

[Quantization of AVC]

Herein, the quantization defined by the AVC (Advanced Video Coding) isdescribed as an example of a conventional quantization process.

Although an integer transform matrix [H] defined by the AVC does notsatisfy a requirement of an orthogonal transform matrix represented by afollowing equation (1), the orthogonal transform process is performed byperforming different quantization processes to respective componentsafter integer transform and combining the integer transform and thequantization.[H][H]^(T) =[I]  (1)

In the AVC, it is possible to define a quantization parameter QP, whichmay have values from “0” to “51”, for each macroblock in order toperform the quantization.

For example, suppose that A(QP) and B(QP) have values, which satisfy afollowing equation (2), regardless of the value of the QP.A(QP)*B(QP)=2m+n  (2)

The orthogonal transform and the inverse orthogonal transform in the AVCmay be realized by operation represented by following equations (3) and(4).d=c*A(QP)/2^(m)  (3)c′=d*B(QP)/2^(n)  (4)

Meanwhile, c represents an orthogonal transform coefficient before thequantization, d represents the orthogonal transform coefficient afterthe quantization, and c′ represents the orthogonal transform coefficientafter the inverse quantization.

By performing such a process, it is possible to realize the quantizationand inverse quantization processes not by division but only by shiftoperation in the AVC.

Meanwhile, values of A and B differ depending on components.

The quantization parameter QP is designed such that the quantizationprocess twice as coarse as the original one is performed when the valuethereof increments by 6 such as from 6 to 12, for example.

Especially, deterioration in a chrominance signal is easily noticeableat a lower bit rate, that is, with a higher QP. Therefore, a defaultquantization parameter QP_(C) for the chrominance signal is defined withrespect to a quantization parameter QP_(Y) for a luminance signal asindicated in a table in FIG. 2.

A user may control this relationship by setting information aboutChromaQPOffset included in image compressed information.

Also, in a profile not lower than a High Profile, it is possible toindependently set the quantization parameter for a Cb/Cr component byusing ChromaQPOffset and 2ndChromaQPOffset.

[Quantization Parameter Calculation]

In the AVC coding system and coding systems disclosed in Non-PatentDocuments 1 and 2, a quantization parameter MB_QP for each macroblock iscalculated in the following manner.

That is, QpBdOffset_(Y) is first calculated from bit_depth_luma_minus8in a sequence parameter set as represented by a following equation (5).QpBdOffset_(Y)=6*bit_depth_luma_minus8  (5)Next, an initial value of the quantization parameter in each picture isspecified by pic_init_qp_minus26 in a picture parameter set.

Next, by slice_qp_delta defined in a slice layer, a quantizationparameter SliceQP_(Y) in the slice is calculated as represented by afollowing equation (6).SliceQPY=26+pic_init_qp_minus26+slice_qp_delta  (6)

Finally, by using mb_qp_delta in a macroblock layer, the quantizationparameter MB_QP for each macroblock is calculated as represented by afollowing equation (7).MB_QP=((MB_QP_(Prev)+mb_qp_delta+52+2*QpBdOffset_(Y))%(52+QpBdOffset_(Y)))−QpBdOffset_(Y)  (7)

Herein, MB_QP_(Prev) represents the quantization parameter for aprevious macroblock.

In the present technology, in addition to this, information aboutsubmb_qp_delta is further included in a sub macroblock layer in imagecompression.

By using this information, the quantization parameter SubMB_QP for eachsub macroblock is calculated as represented by a following equation (8).SubMB_QP=Clip(0,51,MB_QP+submb_qp_delta)  (8)

Herein, Clip(min, max, value) represents a function having a returnvalue as represented by a following equation (9).

$\begin{matrix}\left\lbrack {{Equation}\mspace{14mu} 1} \right\rbrack & \; \\{{{Clip}\left( {\min,\max,{value}} \right)} = \begin{matrix}{\min,} & {{if}\left( {{value} < \min} \right)} \\{\max,} & {{if}\left( {{value} > \max} \right)} \\{{value},} & {otherwise}\end{matrix}} & (9)\end{matrix}$

That is, the quantization parameter SubMB_QP for each sub macroblock iscalculated as represented by a following equation (10). Herein, aminimum quantization parameter defined in advance is minQP and a maximumquantization parameter defined in advance is maxQP.[Equation 2]SUBMB_QP=Clip(minQP,maxQP,MB_QP+submb_qp_delta)  (10)

Meanwhile, when there is no submb_qp_delta in the image compressedinformation, a value thereof is set to “0” and the quantizationparameter for the macroblock is also applied to the sub macroblock.

[Quantization Unit]

FIG. 5 is a block diagram illustrating a detailed configuration exampleof the quantization unit 105 in FIG. 1. As illustrated in FIG. 5, thequantization unit 105 includes a sub macroblock activity buffer 151, aquantization parameter calculation unit 152, and a quantizationprocessor 153.

The sub macroblock activity buffer 151 holds the activity supplied fromthe rate controller 117. Although adaptive quantization based on theactivity as defined in a MPEG-2 Test Model is performed in the AVCcoding system, the rate controller 117 calculates the activity (alsoreferred to as sub macroblock activity) for each sub macroblock. Amethod of calculating the sub macroblock activity is similar to that ina conventional case where the activity is calculated for eachmacroblock.

The sub macroblock activity buffer 151 holds the sub macroblock activitysupplied from the rate controller 117 and supplies the held submacroblock activity to the sub macroblock quantization unit 121 for eachpredetermined amount (for example, an amount of one screen).

The sub macroblock quantization unit 121 calculates the quantizationvalue for each sub macroblock by using the sub macroblock activitysupplied from the sub macroblock activity buffer 151. The quantizationvalue for each sub macroblock may be calculated by a method similar tothat in a case where the quantization value for each macroblock iscalculated from the activity for each macroblock.

After obtaining the quantization value for each sub macroblock, the submacroblock quantization unit 121 supplies the quantization value foreach sub macroblock to the quantization parameter calculation unit 152.

The quantization parameter calculation unit 152 calculates variousquantization parameters by using the quantization value for each submacroblock supplied from the sub macroblock quantization unit 121.

For example, the quantization parameter calculation unit 152 calculatesthe quantization parameters such as pic_init_qp_minus26, slice_qp_delta,and mb_qp_delta. The quantization parameter calculation unit 152 mayobtain the quantization value for each macroblock from the quantizationvalue for each sub macroblock. Therefore, the quantization parametercalculation unit 152 calculates the various quantization parameters toset as in a case of the conventional AVC coding system.

The quantization parameter calculation unit 152 further obtains thequantization parameter submb_qp_delta indicating difference between thequantization parameter MB_QP for each macroblock and the quantizationparameter SubMB_QP for each sub macroblock. It is required to transmitthe quantization parameter for each sub macroblock to a decoding side.It becomes possible to decrease a code amount of the quantizationparameter for each sub macroblock by obtaining a difference value inthis manner. So to speak, the quantization parameter submb_qp_delta is atransmission format of the quantization parameter SubMB_QP. Thequantization parameter SubMB_QP for each sub macroblock may be obtainedby transforming the quantization value for each sub macroblock.Similarly, the quantization parameter MB_QP for each macroblock isobtained by transforming the quantization value for each macroblock. Thequantization parameter calculation unit 152 calculates submb_qp_deltafor each sub macroblock by using the above-described equation (35), forexample.

The quantization parameter calculation unit 152 supplies thequantization value for each sub macroblock to the quantization processor153. Also, the quantization parameter calculation unit 152 suppliescalculated various quantization parameters (specifically,pic_init_qp_minus26, slice_qp_delta, mb_qp_delta and the like) to thelossless coding unit 106 and transmits the same together with a codedstream obtained by coding the image. Meanwhile, as described above, whenthe value of submb_qp_delta is “0”, transmission of submb_qp_delta isomitted. That is, in this case, the quantization parameter other thansubmb_qp_delta is supplied to the lossless coding unit 106.

Further, the quantization parameter calculation unit 152 supplies thequantization value for each sub macroblock also to the inversequantization unit 108.

The quantization processor 153 quantizes the orthogonal transformcoefficient supplied from the orthogonal transformation unit 104 byusing the quantization value for each sub macroblock.

The quantization processor 153 supplies a quantized orthogonal transformcoefficient to the lossless coding unit 106 and the inverse quantizationunit 108.

Meanwhile, the inverse quantization unit 108 inversely quantizes theorthogonal transform coefficient quantized by the above-describedquantization unit 105 by using the sub macroblock inverse quantizationunit 122. In an image decoding apparatus corresponding to the imagecoding apparatus 100 also, a process similar to the inverse quantizationprocess is performed, so that the inverse quantization is described indetail when the image decoding apparatus is described.

In the conventional case such as the AVC coding system, only onequantization parameter may be set for one macroblock. Therefore, in acase where a planar area and an area including texture are mixed in onemacroblock, it is difficult to set the quantization parameterappropriate for both of the areas.

Especially, the larger the size of the macroblock as an extendedmacroblock (extended partial area) proposed in Non-Patent Document 2 andthe like, the higher the possibility that the images having differentcharacteristics are mixed in the area, so that it becomes more difficultto perform the adaptive quantization corresponding to thecharacteristics of each area.

On the other hand, the image coding apparatus 100 may calculate an indexindicating the complexity of the image for each sub macroblock by therate controller 117 and calculate the quantization value for each submacroblock by the sub macroblock quantization unit 121. That is, thequantization processor 153 may perform the quantization process by usingthe quantization value appropriate for each sub macroblock.

According to this, the image coding apparatus 100 may perform thequantization process more suitable for contents of the image.Especially, also in a case where the size of the macroblock is extendedand both of the flat area and the area including the texture areincluded in a single macroblock, the image coding apparatus 100 mayperform an adaptive quantization process suitable for each area toinhibit subjective image quality of the decoded image fromdeteriorating.

For example, in an image 160 illustrated in FIG. 6, a macroblock 161includes only the flat area. Therefore, even when the image codingapparatus 100 performs the quantization process by using a singlequantization parameter for such macroblock 161, there is no particularproblem in image quality.

On the other hand, a macroblock 162 includes both of the flat area and atexture area. In the quantization process by using the singlequantization parameter, it is not possible to perform the adaptivequantization appropriate for both of the flat area and the texture area.Therefore, if the image coding apparatus 100 performs the quantizationprocess by using the single quantization parameter for such macroblock161, the subjective image quality of the decoded image might bedeteriorated.

In such a case also, the image coding apparatus 100 may calculate thequantization value for each sub macroblock as described above, so thatit is possible to perform a more appropriate quantization process toinhibit the subjective image quality of the decoded image fromdeteriorating.

Also, when a total code amount for each picture is likely to overflow inthe accumulation buffer 107, control by the quantization parameter isperformed. Therefore, at that time, by allowing the quantization unit105 to calculate the quantization value for each sub macroblock andperform the quantization as described above, the image coding apparatus100 may control measurement against the overflow in a smaller unit.

Further, when the value of submb_qp_delta is “0”, transmission ofsubmb_qp_delta is omitted, so that it is possible to inhibit unnecessaryreduction in coding efficiency. When the value of submb_qp_delta is “0”,the quantization parameter SubMB_QP for each sub macroblock and thequantization parameter MB_QP for each macroblock are equal to eachother. Therefore, it is possible to make the quantization parameterMB_QP for each macroblock the quantization parameter SubMB_QP for eachsub macroblock on the decoding side, so that the value of submb_qp_delta(“0”) is not required. Therefore, it is possible to omit thetransmission of submb_qp_delta as described above. It goes withoutsaying that submb_ qp_delta having the value of “0” may be transmitted;however, it is possible to improve the coding efficiency by omitting thetransmission of submb_qp_delta.

[Flow of Coding Process]

Next, a flow of each process executed by the above-described imagecoding apparatus 100 is described. First, an example of a flow of acoding process is described with reference to a flowchart in FIG. 7.

At step S101, the A/D converter 101 A/D converts the input image. Atstep S102, the screen reorder buffer 102 stores the A/D converted imageand reorders pictures in order of display into order for coding.

At step S103, the arithmetic unit 103 calculates difference between theimage reordered by the process at step S102 and the predicted image. Thepredicted image is supplied to the arithmetic unit 103 through theselector 116 from the motion prediction/compensation unit 115 in a caseof the inter prediction and from the intra prediction unit 114 in a caseof the intra prediction, respectively.

A data amount of difference data is smaller than that of original imagedata. Therefore, it is possible to compress the data amount as comparedto a case where the image is directly coded.

At step S104, the orthogonal transformation unit 104 orthogonallytransforms the difference information generated by the process at stepS103. Specifically, the orthogonal transform such as the discrete cosinetransform and the Karhunen-Loeve transform is performed and thetransform coefficient is output.

At step S105, the quantization unit 105 and the sub macroblockquantization unit 121 obtain the quantization parameter. A flow of aquantization parameter calculation process is described later in detail.

At step S106, the quantization processor 153 of the quantization unit105 quantizes the orthogonal transform coefficient obtained by theprocess at step S104 by using the quantization value for each submacroblock calculated by the process at step S105.

The difference information quantized by the process at step S106 islocally decoded in a following manner. That is, at step S107, theinverse quantization unit 108 inversely quantizes the quantizedorthogonal transform coefficient (also referred to as a quantizedcoefficient) generated by the process at step S106 by characteristicscorresponding to characteristics of the quantization unit 105. At stepS108, the inverse orthogonal transformation unit 109 inverselyorthogonally transforms the orthogonal transform coefficient obtained bythe process at step S107 by characteristics corresponding tocharacteristics of the orthogonal transformation unit 104.

At step S109, the arithmetic unit 110 adds the predicted image to thelocally decoded difference information to generate the locally decodedimage (image corresponding to an input to the arithmetic unit 103). Atstep S110, the deblocking filter 111 filters the image generated by theprocess at step S109. According to this, the block distortion isremoved.

At step S111, the frame memory 112 stores the image from which the blockdistortion has been removed by the process at step S110. Meanwhile, theimage, which is not subjected to the filter process by the deblockingfilter 111, also is supplied from the arithmetic unit 110 to the framememory 112 to be stored.

At step S112, the intra prediction unit 114 performs an intra predictionprocess in the intra prediction mode. At step S113, the motionprediction/compensation unit 115 performs an inter motion predictionprocess in which motion prediction and motion compensation in the interprediction mode are performed.

At step S114, the selector 116 determines an optimal prediction modebased on each cost function value output from the intra prediction unit114 and the motion prediction/compensation unit 115. That is, theselector 116 selects the predicted image generated by the intraprediction unit 114 or the predicted image generated by the motionprediction/compensation unit 115.

Also, selection information indicating the selected predicted image issupplied to the intra prediction unit 114 or the motionprediction/compensation unit 115 of which predicted image is selected.When the predicted image in the optimal intra prediction mode isselected, the intra prediction unit 114 supplies information indicatingthe optimal intra prediction mode (that is, the intra prediction modeinformation) to the lossless coding unit 106.

When the predicted image of an optimal inter prediction mode isselected, the motion prediction/compensation unit 115 outputsinformation indicating the optimal inter prediction mode and informationcorresponding to the optimal inter prediction mode as necessary to thelossless coding unit 106. The information corresponding to the optimalinter prediction mode includes the motion vector information, flaginformation, reference frame information and the like.

At step S115, the lossless coding unit 106 codes the transformcoefficient quantized by the process at step S106. That is, the losslesscoding such as the variable-length coding and the arithmetic coding isperformed to a difference image (secondary difference image in a case ofinter).

Meanwhile, the lossless coding unit 106 codes the quantization parametercalculated at step S105 to add to the coded data.

Also, the lossless coding unit 106 codes the information about theprediction mode of the predicted image selected by the process at stepS114 to add to the coded data obtained by coding the difference image.That is, the lossless coding unit 106 codes the intra prediction modeinformation supplied from the intra prediction unit 114 or theinformation corresponding to the optimal inter prediction mode suppliedfrom the motion prediction/compensation unit 115 and the like to add tothe coded data.

At step S116, the accumulation buffer 107 accumulates the coded dataoutput from the lossless coding unit 106. The coded data accumulated inthe accumulation buffer 107 is appropriately read to be transmitted tothe decoding side through the transmission channel.

At step S117, the rate controller 117 controls the rate of thequantization operation of the quantization unit 105 such that theoverflow or the underflow does not occur based on the compressed imageaccumulated in the accumulation buffer 107 by the process at step S116.

When the process at step S117 is finished, the coding process isfinished.

[Flow of Quantization Parameter Calculation Process]

Next, an example of the flow of the quantization parameter calculationprocess executed at step S105 in FIG. 7 is described with reference to aflowchart in FIG. 8.

When the quantization parameter calculation process is started, at stepS131, the sub macroblock activity buffer 151 obtains the sub macroblockactivity supplied from the rate controller 117. The sub macroblockactivity buffer 151 holds the obtained sub macroblock activity by theamount of one screen, for example.

At step S132, the sub macroblock quantization unit 121 obtains the submacroblock activity by the amount of one screen, for example, from thesub macroblock activity buffer 151. Then, the sub macroblockquantization unit 121 calculates the quantization value for each submacroblock by using the obtained sub macroblock activity.

At step S133, the quantization parameter calculation unit 152 obtainsthe quantization parameter pic_init_qp_minus26 by using the quantizationvalue for each sub macroblock calculated at step S132.

At step S134, the quantization parameter calculation unit 152 obtainsthe quantization parameter slice_qp_delta by using the quantizationvalue for each sub macroblock calculated at step S132.

At step S135, the quantization parameter calculation unit 152 obtainsthe quantization parameter mb_qp_delta by using the quantization valuefor each sub macroblock calculated at step S132.

At step S136, the quantization parameter calculation unit 152 obtainsthe quantization parameter submb_qp_delta by using the quantizationvalue for each sub macroblock calculated at step S132.

After obtaining the various quantization parameters as described above,the quantization unit 105 finishes the quantization parameter operationprocess, returns the process to step S105 in FIG. 7, and allows theprocess at step S106 and subsequent steps to be executed.

Since the coding process and the quantization parameter calculationprocess are performed in the above-described manner, the image codingapparatus 100 may set the quantization value for each sub macroblock andperform the more appropriate quantization process.

Also, since the quantization parameter calculated in this manner istransmitted to the image coding apparatus, the image coding apparatus100 may allow the image decoding apparatus to obtain the quantizationvalue for each sub macroblock and perform the inverse quantization byusing the same.

2. Second Embodiment

[Image Decoding Apparatus]

FIG. 9 is a block diagram illustrating a principal configuration exampleof an image decoding apparatus to which the present technology isapplied. An image decoding apparatus 200 illustrated in FIG. 9 is adecoding apparatus corresponding to an image coding apparatus 100.

Coded data coded by the image coding apparatus 100 is transmitted to theimage decoding apparatus 200 corresponding to the image coding apparatus100 through a predetermined transmission channel to be decoded.

As illustrated in FIG. 9, the image decoding apparatus 200 includes anaccumulation buffer 201, a lossless decoding unit 202, an inversequantization unit 203, an inverse orthogonal transformation unit 204, anarithmetic unit 205, a deblocking filter 206, a screen reorder buffer207, and a D/A converter 208. The image decoding apparatus 200 alsoincludes a frame memory 209, a selector 210, an intra prediction unit211, a motion prediction/compensation unit 212, and a selector 213.

Further, the image decoding apparatus 200 includes a sub macroblockinverse quantization unit 221.

The accumulation buffer 201 accumulates the transmitted coded data. Thecoded data is coded by the image coding apparatus 100. The losslessdecoding unit 202 decodes the coded data read from the accumulationbuffer 201 at a predetermined timing by a system corresponding to acoding system of a lossless coding unit 106 in FIG. 1.

The inverse quantization unit 203 operates in cooperation with the submacroblock inverse quantization unit 221 to inversely quantizecoefficient data obtained by decoding by the lossless decoding unit 202(quantized coefficient) by a system corresponding to a quantizationsystem of a quantization unit 105 in FIG. 1. That is, the inversequantization unit 203 inversely quantizes the quantized coefficient by amethod similar to that of an inverse quantization unit 108 in FIG. 1 byusing a quantization parameter calculated for each sub macroblocksupplied from the image coding apparatus 100.

The inverse quantization unit 203 supplies the inversely quantizedcoefficient data, that is, an orthogonal transform coefficient to theinverse orthogonal transformation unit 204. The inverse orthogonaltransformation unit 204 inversely orthogonally transforms the orthogonaltransform coefficient by a system corresponding to an orthogonaltransform system of an orthogonal transformation unit 104 in FIG. 1 toobtain decoded residual data corresponding to residual data beforeorthogonal transform by the image coding apparatus 100.

The decoded residual data obtained by inverse orthogonal transform issupplied to the arithmetic unit 205. A predicted image is supplied fromthe intra prediction unit 211 or the motion prediction/compensation unit212 through the selector 213 to the arithmetic unit 205.

The arithmetic unit 205 adds the decoded residual data to the predictedimage to obtain decoded image data corresponding to image data beforesubtraction of the predicted image by an arithmetic unit 103 of theimage coding apparatus 100. The arithmetic unit 205 supplies the decodedimage data to the deblocking filter 206.

The deblocking filter 206 removes block distortion from the supplieddecoded image and thereafter supplies the same to the screen reorderbuffer 207.

The screen reorder buffer 207 reorders an image. That is, framesreordered into order for coding by a screen reorder buffer 102 in FIG. 1are reordered into original order of display. The D/A converter 208 D/Aconverts the image supplied from the screen reorder buffer 207 andoutputs the same to a display (not illustrated) for display.

An output of the deblocking filter 206 is further supplied to the framememory 209.

The frame memory 209, the selector 210, the intra prediction unit 211,the motion prediction/compensation unit 212, and the selector 213correspond to a frame memory 112, a selector 113, an intra predictionunit 114, a motion prediction/compensation unit 115, and a selector 116of the image coding apparatus 100, respectively.

The selector 210 reads an image to which an inter process is performedand a reference image from the frame memory 209 to supply the images tothe motion prediction/compensation unit 212. Also, the selector 210reads an image used for intra prediction from the frame memory 209 tosupply the image to the intra prediction unit 211.

Information indicating an intra prediction mode and the like obtained bydecoding header information is appropriately supplied from the losslessdecoding unit 202 to the intra prediction unit 211. The intra predictionunit 211 generates the predicted image from the reference image obtainedfrom the frame memory 209 based on this information and supplies thegenerated predicted image to the selector 213.

The motion prediction/compensation unit 212 obtains the informationobtained by decoding the header information (prediction modeinformation, motion vector information, reference frame information, aflag, various parameters and the like) from the lossless decoding unit202.

The motion prediction/compensation unit 212 generates the predictedimage from the reference image obtained from the frame memory 209 basedon the information supplied from the lossless decoding unit 202 andsupplies the generated predicted image to the selector 213.

The selector 213 selects the predicted image generated by the motionprediction/compensation unit 212 or the intra prediction unit 211 andsupplies the same to the arithmetic unit 205.

The sub macroblock inverse quantization unit 221 obtains thequantization parameter from the inverse quantization unit 203 andobtains a quantization value for each sub macroblock by using anequation (10) and returns the same to the inverse quantization unit 203.

[Inverse Quantization Unit]

FIG. 10 is a block diagram illustrating a detailed configuration exampleof the inverse quantization unit 203.

As illustrated in FIG. 10, the inverse quantization unit 203 includes aquantization parameter buffer 251, an orthogonal transform coefficientbuffer 252, and an inverse quantization processor 253.

The parameter about quantization in each layer such as a pictureparameter set and a slice header of the coded data supplied from theimage coding apparatus 100 is decoded by the lossless decoding unit 202to be supplied to the quantization parameter buffer 251. Thequantization parameter buffer 251 appropriately holds the quantizationparameter, and supplies the quantization parameter to the sub macroblockinverse quantization unit 221 at a predetermined timing.

The sub macroblock inverse quantization unit 221 calculates aquantization parameter SubMB_QP for each sub macroblock as representedby equations (5) to (10), for example, by using the quantizationparameter supplied from the quantization parameter buffer 251 andtransforms the same to the quantization value for each sub macroblock tosupply the quantization value to the inverse quantization processor 253.

Meanwhile, as described above in the first embodiment, when a value ofsubmb_qp_delta is “0”, submb qp_delta is not transmitted. The submacroblock inverse quantization unit 221 applies a value of aquantization parameter MB_QP for each macroblock to the quantizationparameter SubMB_QP for each sub macroblock when there is nosubmb_qp_delta in the quantization parameter supplied from thequantization parameter buffer 251.

Also, the quantized orthogonal transform coefficient obtained bydecoding the coded data supplied from the image coding apparatus 100 bythe lossless decoding unit 202 is supplied to the orthogonal transformcoefficient buffer 252. The orthogonal transform coefficient buffer 252appropriately holds the quantized orthogonal transform coefficient andsupplies the same to the inverse quantization processor 253 at apredetermined timing.

The inverse quantization processor 253 inversely quantizes the quantizedorthogonal transform coefficient supplied from the orthogonal transformcoefficient buffer 252 by using the quantization value for each submacroblock supplied from the sub macroblock inverse quantization unit221. The inverse quantization processor 253 supplies the orthogonaltransform coefficient obtained by inverse quantization to the inverseorthogonal transformation unit 204.

As described above, the inverse quantization unit 203 may perform aninverse quantization process by using the quantization value calculatedfor each sub macroblock. According to this, the image decoding apparatus200 may perform the inverse quantization process more suitable forcontents of the image. Especially, even in a case where the macroblocksize is extended and both of a flat area and an area including textureare included in a single macroblock, the image decoding apparatus 200may perform an adaptive inverse quantization process suitable for eacharea to inhibit subjective image quality of the decoded image fromdeteriorating.

Meanwhile, the inverse quantization unit 108 of the image codingapparatus 100 illustrated in FIG. 1 also has a configuration similar tothat of the inverse quantization unit 203 and performs a similarprocess. However, the inverse quantization unit 108 obtains thequantization parameter supplied from the quantization unit 105 and thequantized orthogonal transform coefficient and performs the inversequantization.

Also, the inverse quantization unit 108 provides the quantizationparameter to a sub macroblock inverse quantization unit 122, whichperforms a process similar to that of the sub macroblock inversequantization unit 221, and allows the sub macroblock inversequantization unit 122 to generate the quantization value for each submacroblock.

[Flow of Decoding Process]

Next, a flow of each process executed by the above-described imagedecoding apparatus 200 is described. First, an example of a flow of adecoding process is described with reference to a flowchart in FIG. 11.

When the decoding process is started, the accumulation buffer 201accumulates the transmitted coded data at step S201. At step S202, thelossless decoding unit 202 decodes the coded data supplied from theaccumulation buffer 201. That is, an I picture, a P picture, and a Bpicture coded by the lossless coding unit 106 in FIG. 1 are decoded.

At that time, the motion vector information, the reference frameinformation, the prediction mode information (intra prediction mode orinter prediction mode), and the information such as the flag and thequantization parameter are also decoded.

When the prediction mode information is the intra prediction modeinformation, the prediction mode information is supplied to the intraprediction unit 211. When the prediction mode information is the interprediction mode information, the motion vector information correspondingto the prediction mode information is supplied to the motionprediction/compensation unit 212.

At step S203, the inverse quantization unit 203 inversely quantizes thequantized orthogonal transform coefficient obtained by decoding by thelossless decoding unit 202. At step S204, the inverse orthogonaltransformation unit 204 inversely orthogonally transforms the orthogonaltransform coefficient obtained by the inverse quantization by theinverse quantization unit 203 by a method corresponding to theorthogonal transformation unit 104 in FIG. 1. According to this,difference information corresponding to an input of the orthogonaltransformation unit 104 in FIG. 1 (output of the arithmetic unit 103) isdecoded.

At step S205, the arithmetic unit 205 adds the predicted image to thedifference information obtained by the process at step S204. Accordingto this, original image data is decoded.

At step S206, the deblocking filter 206 appropriately filters thedecoded image obtained by the process at step S205. According to this,block distortion is appropriately removed from the decoded image.

At step S207, the frame memory 209 stores the filtered decoded image.

At step S208, the intra prediction unit 211 or the motionprediction/compensation unit 212 performs a prediction process of theimage according to the prediction mode information supplied from thelossless decoding unit 202.

That is, in a case where the intra prediction mode information issupplied from the lossless decoding unit 202, the intra prediction unit211 performs an intra prediction process in the intra prediction mode.Also, in a case where the inter prediction mode information is suppliedfrom the lossless decoding unit 202, the motion prediction/compensationunit 212 performs a motion prediction process in the inter predictionmode.

At step S209, the selector 213 selects the predicted image. That is, thepredicted image generated by the intra prediction unit 211 or thepredicted image generated by the motion prediction/compensation unit 212is supplied to the selector 213. The selector 213 selects the unit ofwhich predicted image is supplied and supplies the predicted image tothe arithmetic unit 205. The predicted image is added to the differenceinformation by the process at step S205.

At step S210, the screen reorder buffer 207 reorders the frames of thedecoded image data. That is, the frames of the decoded image datareordered for the coding by the screen reorder buffer 102 of the imagecoding apparatus 100 (FIG. 1) are reordered into the original order ofdisplay.

At step S211, the D/A converter 208 D/A converts the decoded image dataof which frames are reordered by the screen reorder buffer 207. Thedecoded image data is output to a display not illustrated and the imageis displayed.

[Inverse Quantization Process]

Next, an example of a flow of the inverse quantization process isdescribed with reference to a flowchart in FIG. 12.

When the inverse quantization process is started, the quantizationparameter buffer 251 obtains a quantization parameterpic_init_qp_minus26 supplied from the lossless decoding unit 202 at stepS231.

At step S232, the quantization parameter buffer 251 obtains aquantization parameter slice_qp_delta supplied from the losslessdecoding unit 202.

At step S233, the quantization parameter buffer 251 obtains thequantization parameter mb_qp_delta supplied from the lossless decodingunit 202.

At step S234, the quantization parameter buffer 251 obtains aquantization parameter submb_qp_delta supplied from the losslessdecoding unit 202. However, when there is no submb_qp_delta, the processat step S234 is omitted.

At step S235, the sub macroblock inverse quantization unit 221calculates the quantization value for each sub macroblock by usingvarious quantization parameters obtained by the processes at steps S231to S234. However, when submb_qp_delta is not supplied from the imagecoding apparatus 100 and the process at step S234 is omitted, the submacroblock inverse quantization unit 221 applies the quantization valuefor each macroblock to the quantization value for each sub macroblock.

At step S236, the inverse quantization processor 253 inversely quantizesthe quantized orthogonal transform coefficient held by the orthogonaltransform coefficient buffer 252 by using the quantization value foreach sub macroblock calculated by the process at step S235.

When the process at step S236 is finished, the inverse quantization unit203 returns the process to step S203 and allows the processes at stepS204 and subsequent steps to be executed.

By performing the decoding process and the inverse quantization processas described above, the image decoding apparatus 200 may perform theinverse quantization process by using the quantization value calculatedfor each sub macroblock and perform the inverse quantization processmore suitable for the contents of the image.

3. Third Embodiment

[submb_qp_present_flag]

Although it has been described above that submb_qp_delta isappropriately transmitted as the quantization parameter, it is alsopossible to further transmit a flag, which acknowledges presence ofsubmb_qp_delta for each macroblock.

In this case, a configuration of an image coding apparatus 100 issimilar to a configuration example illustrated in FIG. 1. Also, aconfiguration of a quantization unit 105 is similar to a configurationexample illustrated in FIG. 5. However, a quantization parametercalculation unit 152 further calculates submb_qp_present_flag, which isflag information indicating whether submb_qp_delta of which value is not“0” is present, for each macroblock. When any submb_qp_delta of submacroblocks belonging to the macroblock has the value not “0”,submb_qp_present_flag is set to “1”, for example. Also, whensubmb_qp_delta of all the sub macroblocks belonging to the macroblockare “0”, submb_qp_present_flag is set to “0”, for example.

It goes without saying that a value of submb_qp_present_flag isarbitrary and any value may be used as long as it is possible toidentify a case where any submb_qp_delta has the value not “0” from acase where submb_qp_delta of all the sub macroblocks are “0”.

When the quantization parameter calculation unit 152 sets the value inthis manner, the quantization parameter calculation unit 152 suppliessubmb_qp_present_flag to the lossless coding unit 106 as one of thequantization parameters. The lossless coding unit 106 adds thissubmb_qp_present_flag to a macroblock header, for example, and codes thesame. That is, submb_qp_present_flag is transmitted together with codeddata as well as another quantization parameter.

Therefore, a coding process in this case is performed as in the casedescribed above with reference to the flowchart in FIG. 7. Also, anexample of a flow of a quantization parameter calculation process inthis case is described with reference to a flowchart in FIG. 13. In thiscase also, the quantization parameter calculation process is performedin a manner basically similar to that in the case illustrated withreference to the flowchart in FIG. 8.

That is, processes at steps S331 to S336 are performed as the processesat steps S131 to S136 in FIG. 8. However, in this case, the quantizationparameter calculation unit 152 further calculates the quantizationparameter submb_qp_present_flag at step S337.

As described above, the quantization parameter submb_qp_present_flag iscalculated to be transmitted.

That is, submb_qp_present_flag is present in each macroblock header ofthe data. Then, submb_qp_delta is present in a sub macroblock header ofthe macroblock in which the value of submb_qp_present_flag is “1” andsubmb_qp_delta is not present in the sub macroblock header of themacroblock in which the value of submb_qp_present_flag is “0”.

Such coded data is transmitted from the image coding apparatus 100 to animage decoding apparatus 200.

A configuration of the image decoding apparatus 200 in this case issimilar to a configuration example illustrated in FIG. 9. Also, aconfiguration of an inverse quantization unit 203 is similar to aconfiguration example illustrated in FIG. 10. However, a sub macroblockinverse quantization unit 221 calculates a quantization value for eachmacroblock for the macroblock in which submb_qp_present_flag is set to“0” without waiting for supply of submb_qp_delta and applies thequantization value to the quantization value for each sub macroblock.

In other words, the sub macroblock inverse quantization unit 221 obtainssubmb_qp_delta only when submb_qp_present_flag is “1” and calculates thequantization value for each sub macroblock.

A decoding process in this case is performed in a manner similar to thatdescribed above with reference to the flowchart in FIG. 11. Also, anexample of a flow of an inverse quantization process in this case isdescribed with reference to a flowchart in FIG. 14. In this case also,the inverse quantization process is performed in a manner basicallysimilar to that in the case described with reference to the flowchart inFIG. 12.

That is, processes at steps S431 to S433 are performed as the processesat steps S231 to S233 in FIG. 12. However, in this case, thequantization parameter buffer 251 obtains the quantization parametersubmb_qp_present_flag stored in the macroblock header at step S434.

At step S435, the sub macroblock inverse quantization unit 221determines whether the value of the quantization parametersubmb_qp_present_flag is “1”. When the value of the quantizationparameter submb_qp_present_flag is “1”, the quantization parameterbuffer 251 obtains the quantization parameter submb_qp_delta at stepS436. At step S437, the sub macroblock inverse quantization unit 221calculates the quantization value for each sub macroblock. That is,processes similar to those at steps S234 and S235 in FIG. 12 areperformed.

Also, when it is determined that the value of the quantization parametersubmb_qp_present_flag is “0” at step S435, the sub macroblock inversequantization unit 221 calculates the quantization value for eachmacroblock at step S438 and applies the same as the quantization valuefor each sub macroblock.

When the quantization value is calculated as in the above-describedmanner, the inverse quantization processor 253 performs inversequantization by using the quantization value at step S439.

As described above, the image decoding apparatus 200 may more easilygrasp the presence of the quantization parameter submb_qp_delta and maymore easily calculate the quantization value without need for anunnecessary process of searching submb_qp_delta, which is not present,by transmitting submb_qp_present_flag indicating the presence of thequantization parameter submb_qp_delta for each macroblock to use at thetime of the inverse quantization.

Although the image coding apparatus, which codes by the systemequivalent to the AVC, and the image decoding apparatus, which decodesby the system equivalent to the AVC, are described above as an examplein the first to third embodiments, the scope of application of thepresent technology is not limited thereto and the present technology maybe applied to every image coding apparatus and image decoding apparatus,which perform the coding process based on a block having a hierarchicalstructure as illustrated in FIG. 4.

Also, the above-described various quantization parameters may be addedto an arbitrary position of the coded data or may be transmitted to adecoding side separately from the coded data, for example. For example,the lossless coding unit 106 may describe the information in a bitstream as syntax. Also, the lossless coding unit 106 may store theinformation in a predetermined area as auxiliary information totransmit. For example, the information may be stored in a parameter set(for example, header and the like of sequence and picture) such as SEI(Supplemental Enhancement Information).

It is also possible for the lossless coding unit 106 to transmit theinformation from the image coding apparatus to the image decodingapparatus separately from the coded data (as another file). In thiscase, it is necessary to clarify correspondence relationship between theinformation and the coded data (such that the decoding side may graspthe relationship) but any method may be used therefor. For example, itis possible to separately create table information indicating thecorrespondence relationship or to embed link information indicatingcorresponding data in each data.

Meanwhile, it is also possible that the above-described quantizationusing the quantization value for each sub macroblock (calculation of thequantization parameter for each sub macroblock) is performed only for anextended macroblock not smaller than 32×32.

For example, a rate controller 117 calculates activity for each submacroblock only when the current macroblock is the extended macroblockand calculates the activity for each macroblock when the currentmacroblock is a conventional macroblock not larger than 16×16 defined inan existing coding standard such as the AVC.

The sub macroblock quantization unit 121 calculates the quantizationvalue for each sub macroblock only for the extended macroblock andcalculates the quantization value for each macroblock for theconventional macroblock not larger than 16×16, for example.

The quantization parameter calculation unit 152 calculates thequantization parameter Submb_qp_delta only for the extended macroblockand does not calculate the quantization parameter submb_qp_delta for theconventional macroblock not larger than 16×16, for example.

The quantization processor 153 performs the quantization by using thequantization value for each sub macroblock only for the extendedmacroblock and performs the quantization by using the quantization valuefor each macroblock for the conventional macroblock not larger than16×16, for example.

In the above-described manner, the image coding apparatus 100 mayperform the quantization by using the quantization value for each submacroblock only for the extended macroblock having a large area in whichan effect of inhibiting deterioration in subjective image quality of adecoded image may be sufficiently expected and perform the quantizationby using the quantization value for each macroblock for the macroblockhaving a conventional size in which the expectation for the effect isrelatively small. According to this, the image coding apparatus 100 mayinhibit increase in load caused by the quantization using thequantization value for each sub macroblock.

In this case, it is of course possible for the image decoding apparatus200 to perform the inverse quantization by using the quantization valuefor each sub macroblock only for the extended macroblock as the imagecoding apparatus 100.

For example, the sub macroblock inverse quantization unit 221 calculatesthe quantization value for each sub macroblock only for the extendedmacroblock and calculates the quantization value for each macroblock forthe conventional macroblock not larger than 16×16.

Therefore, the inverse quantization processor 253 performs the inversequantization by using the quantization value for each sub macroblockonly for the extended macroblock and performs the inverse quantizationby using the quantization value for each macroblock for the conventionalmacroblock not larger than 16×16, for example.

In the above-described manner, the image decoding apparatus 200 mayperform the inverse quantization by using the quantization value foreach sub macroblock only for the extended macroblock having the largearea in which the effect of inhibiting the deterioration in thesubjective image quality of the decoded image may be sufficientlyexpected and perform the inverse quantization by using the quantizationvalue for each macroblock for the macroblock having the conventionalsize in which the expectation for the effect is relatively small.According to this, the image decoding apparatus 200 may inhibit theincrease in the load caused by the inverse quantization using thequantization value for each sub macroblock.

Meanwhile, when submb_qp_present_flag is transmitted as in the thirdembodiment, it may be configured to transmit the quantization parametersubmb_qp_present_flag only for the extended macroblock. In other words,transmission of the quantization parameter submb_qp_present_flag may beomitted for the macroblock having the conventional size. Of course itmay be configured to transmit the quantization parametersubmb_qp_present_flag having the value indicating that there is noquantization parameter submb_qp_delta of which value is other than “0”for the macroblock having the conventional size.

4. Fourth Embodiment

[Summary]

Although it has been described above that the quantization parameter isspecified for each sub macroblock, a manner of assigning thequantization parameter to the sub macroblock may be other than theabove-described one. For example, it is also possible to define aquantization parameter SubMB_QP assigned to each sub macroblock asrepresented by a following equation (11) by using a quantizationparameter submb_qp_delta for each sub macroblock and a quantizationparameter previous_qp for sub macroblock coded immediately before thesame.SubMB_QP=Clip(0,51,previous_qp+submb_qp_delta)  (11)

[Coding Unit]

Such a method is to be described below; it is hereinafter described byusing a unit referred to as a coding unit in place of theabove-described macroblock and sub macroblock.

For example, in a “Test Model Under Consideration” (JCTVC-B205), anextended macroblock described with reference to FIG. 4 is defined by aconcept referred to as the coding unit.

The coding unit is a division unit of an image (one picture), which is aunit of process such as a coding process of image data. That is, thecoding unit is a block (partial area) obtained by dividing the image(one picture) into a plurality of parts. That is, the coding unitcorresponds to the above-described macroblock and sub macroblock.

FIG. 15 is a view illustrating a configuration example of the codingunit. As illustrated in FIG. 15, an area of the coding unit may befurther divided into a plurality of parts and each area may be made thecoding unit of one layer lower. That is, the coding units may behierarchically configured (configured to have a tree structure). Inaddition, a size of the coding unit is arbitrary and the coding unitshaving different sizes may be present in one picture.

In an example in FIG. 15, the size of the coding unit in a highest layer(depth=0) is set to 128×128 pixels, an area of 64×64 pixels obtained bydividing the same in half vertically and horizontally (into four) ismade the coding unit in one layer lower (depth=1), and hierarchizationof the coding units is similarly repeated and an area of 8×8 pixels ismade the coding unit in a lowest layer (depth=4).

At that time, the coding unit in the highest layer is referred to as anLCU (Largest Coding Unit) and the coding unit of the lowest layer isreferred to as a SCU (Smallest Coding Unit). That is, the LCUcorresponds to the macroblock and the coding unit in the lower layercorresponds to the sub macroblock.

Meanwhile, the size and a shape of the coding unit of each layer and thenumber of layers are arbitrary. That is, it is not required that thesizes and the shapes of all the LCU and SCU be the same in the image(one picture), the number of layers of the coding unit may be differentaccording to a position in the image, and a manner of dividing the areais also arbitrary. That is, the tree structure of the coding units maybe an arbitrary structure.

It goes without saying that a degree of freedom of the hierarchicalstructure of the coding units may be partially limited such that themanners of dividing the area are the same but only the numbers of layersare different, for example. For example, as illustrated in FIG. 15, itis possible to configure such that one area (one picture or one codingunit) is divided in half vertically and horizontally (that is, intofour) in any position and the sizes of the LCU and SCU in each positionare defined, thereby defining the hierarchical structure of the codingunits.

The sizes of the LCU and SCU may be specified by a sequence parameterset in image compressed information, for example. It goes without sayingthat they may be specified by another metadata and the like.

[Assignment of Quantization Parameter]

In this embodiment, the quantization parameter submb_qp_delta isassigned to each coding unit in place of the macroblock and the submacroblock. However, in this case, the quantization parametersubmb_qp_delta is not a difference value between a quantizationparameter MB_QP for each macroblock and the quantization parameterSubMB_QP for each sub macroblock but the difference value between thequantization parameter previous_qp for a previously coded coding unitand the quantization parameter SubMB_QP for the current coding unit.

In other words, the quantization parameter submb_qp_delta indicating thedifference value between the quantization parameter previous_qp used forprevious coding and the quantization parameter SubMB_QP for the currentcoding unit is assigned to each coding unit. That is, the quantizationparameter submb_qp_delta satisfying the above-described equation (11) isassigned to each coding unit.

Meanwhile, it is only required that an entire area of the image bequantized, so that the quantization parameter submb_qp_delta is actuallyassigned to a part of the coding units such as only to the SCU, forexample.

As in the above-described other embodiments, it is possible to obtainthe quantization parameter SubMB_QP for the current coding unit bytransforming a quantization value obtained from activity for the codingunit. Therefore, the quantization parameter submb_qp_delta for eachcoding unit may be calculated by using the equation (11).

FIG. 16 illustrates the configuration example of the coding unit in oneLCU and an example of the quantization parameter assigned to each codingunit. As illustrated in FIG. 16, a difference value ΔQP between thequantization parameter previous_qp used for the previous coding and thequantization parameter SubMBQP for the current coding unit is assignedto each coding unit (CU) as the quantization parameter.

More specifically, a quantization parameter ΔQP₀ is assigned to an upperleft coding unit 0 (Coding Unit 0) in the LCU. Also, a quantizationparameter ΔQP₁₀ is assigned to an upper left coding unit 10 (Coding Unit10) out of four upper right coding units in the LCU. Further, aquantization parameter ΔQP₁₁ is assigned to an upper right coding unit11 (Coding Unit 11) out of the four upper right coding units in the LCU.Also, a quantization parameter ΔQP₁₂ is assigned to a lower left codingunit 12 (Coding Unit 12) out of the four upper right coding units in theLCU. Further, a quantization parameter ΔQP₁₃ is assigned to a lowerright coding unit 13 (Coding Unit 13) out of the four upper right codingunits in the LCU.

A quantization parameter ΔQP₂₀ is assigned to an upper left coding unit20 (Coding Unit 20) out of four lower left coding units in the LCU.Further, a quantization parameter ΔQP₂₁ is assigned to an upper rightcoding unit 21 (Coding Unit 21) out of the four lower left coding unitsin the LCU. Also, a quantization parameter ΔQP₂₂ is assigned to a lowerleft coding unit 22 (Coding Unit 22) out of the four lower left codingunits in the LCU. Further, a quantization parameter ΔQP₂₃ is assigned toa lower right coding unit 23 (Coding Unit 23) out of the four lower leftcoding units in the LCU. A quantization parameter ΔQP₃ is assigned to alower right coding unit 3 (Coding Unit 3) in the LCU.

The quantization parameter for the coding unit processed immediatelybefore the LCU is set to PrevQP. Further, suppose that the upper leftcoding unit 0 (Coding Unit 0) in the LCU is the current coding unitfirst processed in the LCU.

A quantization parameter CurrentQP for the current coding unit iscalculated as represented by a following equation (12).CurrentQP=PrevQP+ΔQP₀  (12)

Suppose that the coding unit to be processed after the coding unit 0 isthe upper left coding unit 10 (Coding Unit 10) out of the upper rightfour coding units in the LCU illustrated in FIG. 16.

When the coding unit 10 becomes the processing target, the quantizationparameter CurrentQP of the current coding unit is calculated asrepresented by following equations (13) and (14).PrevQP=CurrentQP  (13)CurrentQP=PrevQP+ΔQP₁₀  (14)

In this manner, by making the quantization parameter assigned to eachcoding unit the difference value between the quantization parameter forthe previously coded coding unit and the current quantization parameter,it is not necessary to calculate the quantization parameter for eachmacroblock, so that a quantization process may be performed more easily.

Meanwhile, when the difference value between the quantization parameterfor the already coded coding unit and the current quantization parameteris calculated, it is also possible to calculate the difference valuefrom the coding unit coded before the current coding unit (coding unitcoded before the previously coded coding unit in the LCU). However, thedifference value between the quantization parameter for the previouslycoded coding unit and the current quantization parameter is preferable.

That is, when the difference value between the quantization parameterfor the previously coded coding unit and the current quantizationparameter is calculated, it is only required that only the quantizationparameter for the previously coded coding unit be stored in a memory andthe quantization parameter may be managed in a FIFO (First In First Out)system. Therefore, when the difference value of the quantizationparameter is calculated, the quantization parameter is easily managedand a used amount of memory is small, so that there is an advantage inmounting.

Meanwhile, such a quantization parameter cu_qp_delta for each codingunit is defined by syntax of the coding unit as illustrated in FIG. 17,for example, to be transmitted to a decoding side. That is, thequantization parameter cu_qp_delta for each coding unit corresponds tothe above-described quantization parameter sub_qp_delta.

[Image Coding Apparatus]

FIG. 18 is a block diagram illustrating a principal configurationexample of an image coding apparatus to which the present technology isapplied. An image coding apparatus 300 illustrated in FIG. 18 assignsthe quantization parameter cu_qp_delta to each coding unit as describedabove.

As illustrated in FIG. 18, the image coding apparatus 300 has aconfiguration basically similar to that of an image coding apparatus 100in FIG. 1. However, the image coding apparatus 300 includes a codingunit quantization unit 305 and a rate controller 317 in place of aquantization unit 105, a rate controller 117, and a sub macroblockquantization unit 121 of the image coding apparatus 100. Also, the imagecoding apparatus 300 includes a coding unit inverse quantization unit308 in place of an inverse quantization unit 108 and a sub macroblockinverse quantization unit 122 of the image coding apparatus 100.

The rate controller 317 controls a rate of quantization operation of thecoding unit quantization unit 305 such that overflow or underflow doesnot occur based on a compressed image accumulated in an accumulationbuffer 107. Further, the rate controller 317 provides informationindicating complexity of the image for each coding unit to the codingunit quantization unit 305. The coding unit quantization unit 305performs quantization for each coding unit by using the activity. Also,the coding unit quantization unit 305 calculates the quantizationparameter for each coding unit. The coding unit quantization unit 305supplies an orthogonal transform coefficient (coefficient data)quantized for each coding unit and the calculated quantization parameterfor each coding unit to the lossless coding unit 106 and codes the sameto transmit. Further, the coding unit quantization unit 305 alsoprovides the orthogonal transform coefficient (coefficient data)quantized for each coding unit and the calculated quantization parameterfor each coding unit also to the coding unit inverse quantization unit308.

The coding unit inverse quantization unit 308 performs inversequantization for each coding unit by using the quantization parameterfor each coding unit supplied from the coding unit quantization unit305. The coding unit inverse quantization unit 308 supplies theorthogonal transform coefficient (coefficient data) inversely quantizedfor each coding unit to the inverse orthogonal transformation unit 109.The coding unit inverse quantization unit 308 is to be described laterin detail in description of an image decoding apparatus.

[Detailed Configuration about Quantization]

FIG. 19 is a block diagram illustrating a detailed configuration exampleof the rate controller 317 and the coding unit quantization unit 305.

As illustrated in FIG. 19, the rate controller 317 includes an activitycalculation unit 321 and an activity buffer 322.

The activity calculation unit 321 obtains the image being a target ofthe coding process (current coding unit) from a screen reorder buffer102 and calculates the activity being information indicating dispersionof pixel values as information indicating the complexity of the image.That is, the activity calculation unit 321 calculates the activity foreach coding unit. Meanwhile, it is only required that the quantizationprocess be performed for an entire image, so that it is also possiblethat the activity is calculated only for a part of the coding units suchas only for the SCU, for example.

The activity buffer 322 holds the activity for each coding unitcalculated by the activity calculation unit 321 and provides the same tothe quantization unit 105 at a predetermined timing. The activity buffer322 holds the obtained activity for each coding unit by an amount of onescreen, for example.

A method of calculating the activity is arbitrary and may be a methodsimilar to that of the above-described MPEG2 Test Model, for example.Also, contents of the information indicating the complexity of the imagealso are arbitrary and may be the information other than such activity.

The coding unit quantization unit 305 includes a coding unitquantization value calculation unit 331, a picture quantizationparameter calculation unit 332, a slice quantization parametercalculation unit 333, a coding unit quantization parameter calculationunit 334, and a coding unit quantization unit 335.

The coding unit quantization value calculation unit 331 calculates thequantization value for each coding unit based on the activity for eachcoding unit (information indicating the complexity of the image for eachcoding unit) supplied from the rate controller 317. The quantizationvalue for each coding unit may be calculated by a method similar to thatin a case where the quantization value for each LCU is calculated fromthe activity for each LCU. Meanwhile, it is only required that thequantization process be performed for the entire image, so that it isalso possible that the quantization value for each coding unit iscalculated only for a part of the coding units. Hereinafter, it isassumed that the quantization value for each coding unit is calculatedonly for the SCU as an example.

After obtaining the quantization value for each coding unit, the codingunit quantization value calculation unit 331 supplies the quantizationvalue for each coding unit to the picture quantization parametercalculation unit 332.

The picture quantization parameter calculation unit 332 obtains aquantization parameter pic_init_qp_minus26 for each picture by using thequantization value for each coding unit.

The slice quantization parameter calculation unit 333 obtains aquantization parameter slice_qp_delta for each slice by using thequantization value for each coding unit.

The coding unit quantization parameter calculation unit 334 obtains aquantization parameter cu_qp_delta for each coding unit by using thequantization parameter prevQP used for the previous coding.

The quantization parameters generated by the picture quantizationparameter calculation unit 332 to the coding unit quantization parametercalculation unit 334 are supplied to the lossless coding unit 106,coded, and transmitted to the decoding side, and supplied also to thecoding unit inverse quantization unit 308.

The coding unit quantization unit 335 quantizes the orthogonal transformcoefficient of the current coding unit by using the quantization valuefor each coding unit.

The coding unit quantization unit 335 supplies the orthogonal transformcoefficient quantized for each coding unit to the lossless coding unit106 and the coding unit inverse quantization unit 308.

[Flow of Coding Process]

The image coding apparatus 300 performs the coding process basically asin the case of the image coding apparatus 100 in FIG. 1 described withreference to FIG. 6.

[Flow of Quantization Parameter Calculation Process]

An example of a flow of a quantization parameter calculation processexecuted in the coding process is described with reference to aflowchart in FIG. 20.

When the quantization parameter calculation process is started, at stepS531, the coding unit quantization value calculation unit 331 obtainsthe activity for each coding unit supplied from the rate controller 317.

At step S532, the coding unit quantization value calculation unit 331calculates the quantization value for each coding unit by using theactivity for each coding unit.

At step S533, the picture quantization parameter calculation unit 332obtains the quantization parameter pic_init_qp_minus26 by using thequantization value for each coding unit calculated at step S532.

At step S534, the slice quantization parameter calculation unit 333obtains the quantization parameter slice_qp_delta by using thequantization value for each coding unit calculated at step S532.

At step S535, the coding unit quantization parameter calculation unit334 obtains the quantization parameter cu_qp_delta for each coding unit(ΔQP₀ to ΔQP₂₃ and the like in FIG. 16) by using the quantizationparameter prevQP used for the previous coding.

After obtaining the various quantization parameters in theabove-described manner, the coding unit quantization unit 305 finishesthe quantization parameter calculation process and performs subsequentprocesses of the coding process.

Since the coding process and the quantization parameter calculationprocess are performed in the above-described manner, the image codingapparatus 300 may set the quantization value for each coding unit andperform a more appropriate quantization process according to contents ofthe image.

Also, since the quantization parameter calculated in this manner istransmitted to the image decoding apparatus, the image coding apparatus300 may allow the image decoding apparatus to perform the inversequantization for each coding unit.

Meanwhile, the coding unit inverse quantization unit 308 included in theimage coding apparatus 300 performs a process similar to that of thecoding unit inverse quantization unit included in the image decodingapparatus corresponding to the image coding apparatus 300. That is, theimage coding apparatus 300 may also perform the inverse quantization foreach coding unit.

[Image Decoding Apparatus]

FIG. 21 is a block diagram illustrating a principal configurationexample of the image decoding apparatus to which the present technologyis applied. An image decoding apparatus 400 illustrated in FIG. 21,which corresponds to the above-described image coding apparatus 300,correctly decodes a coded stream (coded data) generated by the coding ofthe image data by the image coding apparatus 300 to generate a decodedimage.

As illustrated in FIG. 21, the image decoding apparatus 400 has aconfiguration basically similar to that of an image decoding apparatus200 in FIG. 8 and performs a similar process. However, the imagedecoding apparatus 400 includes a coding unit inverse quantization unit403 in place of an inverse quantization unit 203 and a sub macroblockinverse quantization unit 221 of the image decoding apparatus 200.

The coding unit inverse quantization unit 403 inversely quantizes theorthogonal transform coefficient quantized for each coding unit by theimage coding apparatus 300 by using the quantization parameter and thelike for each coding unit supplied from the image coding apparatus 300.

FIG. 22 is a block diagram illustrating a principal configurationexample of the coding unit inverse quantization unit 403. As illustratedin FIG. 22, the coding unit inverse quantization unit 403 includes aquantization parameter buffer 411, an orthogonal transform coefficientbuffer 412, a coding unit quantization value calculation unit 413, and acoding unit inverse quantization processor 414.

The quantization parameter in each layer such as a picture parameter setand a slice header of the coded data supplied from the image codingapparatus 300 is decoded by a lossless decoding unit 202 to be suppliedto the quantization parameter buffer 411. The quantization parameterbuffer 411 appropriately holds the quantization parameter and suppliesthe same to the coding unit quantization value calculation unit 413 at apredetermined timing.

The coding unit quantization value calculation unit 413 calculates thequantization value for each coding unit as represented by equations (36)to (39), for example, by using the quantization parameter supplied fromthe quantization parameter buffer 411 and supplies the quantizationvalue to the coding unit inverse quantization processor 414.

Also, the quantized orthogonal transform coefficient obtained bydecoding of the coded data supplied from the image coding apparatus 300by the lossless decoding unit 202 is supplied to the orthogonaltransform coefficient buffer 412. The orthogonal transform coefficientbuffer 412 appropriately holds the quantized orthogonal transformcoefficient to supply to the coding unit inverse quantization processor414 at a predetermined timing.

The coding unit inverse quantization processor 414 inversely quantizesthe quantized orthogonal transform coefficient supplied from theorthogonal transform coefficient buffer 412 by using the quantizationvalue for each coding unit supplied from the coding unit quantizationvalue calculation unit 413. The coding unit inverse quantizationprocessor 414 supplies the orthogonal transform coefficient obtained bythe inverse quantization to an inverse orthogonal transformation unit204.

As described above, the coding unit inverse quantization unit 403 mayperform an inverse quantization process by using the quantization valuecalculated for each coding unit. According to this, the image decodingapparatus 400 may perform the inverse quantization process more suitablefor the contents of the image. Especially, even in a case where a sizeof the macroblock is extended (the size of the LCU is large) and both ofa flat area and an area including texture are included in a single LCU,the image decoding apparatus 400 may perform an adaptive inversequantization process suitable for each area to inhibit subjective imagequality of the decoded image from deteriorating.

Meanwhile, the coding unit inverse quantization unit 308 of the imagecoding apparatus 300 illustrated in FIG. 18 also has a configurationsimilar to that of the coding unit inverse quantization unit 403 andperforms a similar process. However, the coding unit inversequantization unit 308 obtains the quantization parameter and thequantized orthogonal transform coefficient supplied from the coding unitquantization unit 305 and performs the inverse quantization.

[Flow of Decoding Process]

The image decoding apparatus 400 performs a decoding process in a mannerbasically similar to that in a case of the image decoding apparatus 200in FIG. 8 described with reference to the flowchart in FIG. 10.

[Flow of Inverse Quantization Process]

An example of a flow of the inverse quantization process executed in thedecoding process by the image decoding apparatus 400 is described withreference to a flowchart in FIG. 23.

When the inverse quantization process is started, the quantizationparameter buffer 411 obtains the quantization parameterpic_init_qp_minus26 supplied from the lossless decoding unit 202 at stepS631.

At step S632, the quantization parameter buffer 411 obtains thequantization parameter slice_qp_delta supplied from the losslessdecoding unit 202.

At step S633, the quantization parameter buffer 411 obtains thequantization parameter cu_qp_delta supplied from the lossless decodingunit 202.

At step S634, the coding unit quantization value calculation unit 413calculates the quantization value for each coding unit by using thevarious quantization parameters obtained by the processes at steps S631to S633 and the previously used quantization parameter PrevQP.

At step S635, the coding unit inverse quantization processor 414inversely quantizes the quantized orthogonal transform coefficient heldby the orthogonal transform coefficient buffer 412 by using thequantization value for each coding unit calculated by the process atstep S634.

When the process at step S635 is finished, the coding unit inversequantization unit 403 returns the process to the decoding process andallows subsequent processes to be executed.

As described above, by performing the decoding process and the inversequantization process, the image decoding apparatus 400 may perform theinverse quantization process by using the quantization value calculatedfor each coding unit and perform the inverse quantization process moresuitable for the contents of the image.

As described above, in order to decrease a code amount of thequantization parameter for each coding unit (sub macroblock), adifference value dQP between a predetermined quantization parameter andthe quantization parameter SubMB_QP (quantization parametersubmb_qp_delta) is obtained to be transmitted instead of transmittingthe quantization parameter SubMB_QP itself. Two methods represented byfollowing equations (15) and (16) have been described above as methodsof calculating the quantization parameter dQP.dQP=CurrentQP−LCUQP  (15)dQP=CurrentQP−PreviousQP  (16)

In the equations (15) and (16), CurrentQP represents the quantizationparameter for the current coding unit (CU). Also, LCUQP represents thequantization parameter for the LCU to which the current CU belongs (thatis, the current LCU). Further, PreviousQP represents the quantizationparameter for the CU processed immediately before the current CU.

That is, in a case of the equation (15), the difference value betweenthe quantization parameter for the current LCU and the quantizationparameter for the current CU is transmitted. Also, in a case of theequation (16), the difference value between the quantization parameterof the previously processed CU and the quantization parameter of thecurrent CU is transmitted.

The method of calculating such quantization parameter dQP fortransmission is arbitrary and may be other than the above-described twoexamples.

For example, it is also possible to transmit the difference valuebetween a quantization parameter SliceQP for the slice to which thecurrent CU belongs (that is, the current slice) and the quantizationparameter for the current CU as represented by a following equation(17).dQP=CurrentQP−SliceQP  (17)

The quantization parameter CurrentQP may be obtained by transformationof the quantization value of the current CU calculated by the codingunit quantization value calculation unit 331 by the coding unitquantization parameter calculation unit 334 in FIG. 19, for example.Also, the quantization parameter SliceQP may be obtained by the slicequantization parameter calculation unit 333 in FIG. 19 using thequantization parameter pic_init_qp_minus26 obtained by the picturequantization parameter calculation unit 332 and the quantizationparameter slice_qp_delta obtained by itself, for example.

Therefore, for example, the coding unit quantization parametercalculation unit 334 in FIG. 19 may obtain the quantization parameterdQP by using the values. The coding unit quantization parametercalculation unit 334 supplies the quantization parameter dQP to thelossless coding unit 106 to transmit to the decoding side.

The quantization parameter pic_init_qp_minus26 and the quantizationparameter slice_qp_delta are defined in the “Test Model UnderConsideration” (JCTVC-B205), for example, and may be set by a methodsimilar to that of a conventional coding system.

On the decoding side, the quantization parameter for the CU may beobtained from the quantization parameter dQP transmitted from a codingside.

For example, the coding unit quantization value calculation unit 413obtains the quantization parameter SubMB_QP for the CU as represented bya following equation (18) from the quantization parameter dQP andtransforms the same to obtain the quantization value.SubMBQP=Clip(minQP,maxQP,SliceQP+Submb_qp_delta)  (18)

In the equation (18), minQP represents a minimum quantization parameterdefined in advance and maxQP represents a maximum quantization parameterdefined in advance.

In this manner, in a case where the quantization parameter SliceQP isused for obtaining the quantization parameter dQP also, the quantizationand the inverse quantization may be performed as the above-described twomethods. That is, not only the quantization and the inverse quantizationmore suitable for the contents of the image may be performed, but alsothe code amount of the quantization parameter may be decreased.

A table in which characteristics of the processes of the methods arecompared to each other is illustrated in FIG. 24. In the tableillustrated in FIG. 24, a method on the top (referred to as a firstmethod) is a method of obtaining the quantization parameter dQP by usingthe quantization parameter for the LCU. A second top method (referred toas a second method) is a method of obtaining the quantization parameterdQP by using the quantization parameter for the CU processed immediatelybefore the current CU. A method on the bottom (referred to a as thirdmethod) is a method of obtaining the quantization parameter dQP by usingthe quantization parameter for the current slice.

In the table in FIG. 24, easiness of a pipeline process and codingefficiency are compared to each other as the characteristics of themethods. As indicated in the table in FIG. 24, the pipeline process iseasier in the first method than in the second method. The pipe lineprocess is easier in the third method than in the first method. Further,the coding efficiency is better in the first method than in the thirdmethod. The coding efficiency is better in the second method than in thefirst method.

That is, in general, the closer the area is to the current area, thehigher the correlativity with the current area (such as the coding unitand the sub macroblock). Therefore, it is possible to further improvethe coding efficiency of the quantization parameter dQP by obtaining thequantization parameter dQP by using the area closer to the current area.

However, in general, the farther the area is from the current area, theearlier this is processed. Therefore, time until the current area isprocessed becomes longer. That is, allowed time for processing delay andthe like becomes longer. Therefore, when the quantization parameter dQPis obtained by using the area farther from the current area, delay isless likely to occur, which is advantageous for the pipeline process.

As described above, the methods have different characteristics, so thatan appropriate method differs depending on a condition having priority.Meanwhile, it is also possible that each method may be selected. Aselecting method is arbitrary. For example, a user and the like maydetermine in advance the method to be applied. For example, it is alsopossible that any method is adaptively selected according to anarbitrary condition (for each arbitrary unit of process or when anarbitrary event occurs, for example).

When any method is adaptively selected, it is also possible to generateflag information indicating the selected method and transmit the flaginformation from the coding side (quantization side) to the decodingside (inverse quantization side). In this case, the decoding side(inverse quantization side) may select the same method as that of thecoding side (quantization side) by referring to the flag information.

Also, the method of calculating the quantization parameter dQP isarbitrary and may be other than the above-described method. The numberof prepared calculating methods is also arbitrary. Also, the value maybe variable. It is also possible to transmit information defining thequantization parameter dQP from the coding side (quantization side) tothe decoding side (inverse quantization side).

The method of calculating the difference value of the quantizationparameter is illustrated in consideration of the characteristics of theabove-described methods. FIG. 25 illustrates an example ofconfigurations of the LCU and CU. The (number) indicates order of coding(decoding) process of the coding units.

In an LCU(0), the order of coding of the coding units is as follows:

-   -   CU(0)    -   →CU(10)→CU(11)→CU(12)→CU(13)    -   →CU(20)→CU(21)    -   →CU (30)→CU(31)→CU(32)→CU(33)    -   →CU(23)    -   →CU(3)

In this case, the difference value of the quantization parameter is asfollows:

The coding unit CU(0) at the head of the LCU transmits the differencevalue between the quantization parameter SliceQP for the slice to whichthe CU(0) belongs (that is, the current slice) and the quantizationparameter for the current CU(0) by using the equation (17).dQP(CU(0))=CurrentQP(CU0)−SliceQP

Next, the coding units CU(10) to CU(3) other than the one at the head ofthe LCU transmit the difference value between the quantization parameter(CurrentCU) for the current CU and the previously coded CU (PrevioussCU)by using the equation (16).dQP=CurrentQP(CUi)−PreviousQP(CUi−1)

That is, when it is described with reference to FIG. 25, the differencevalues of the quantization parameter are as follows:dQP(CU(10))=CurrentQP(CU(10))−PreviousQP(CU(0))dQP(CU(11))=CurrentQP(CU(11))−PreviousQP(CU(10))dQP(CU(12))=CurrentQP(CU(12))−PreviousQP(CU(11))dQP(CU(13))=CurrentQP(CU(13))−PreviousQP(CU(12))dQP(CU(20))=CurrentQP(CU(20))−PreviousQP(CU(13))dQP(CU(21))=CurrentQP(CU(21))−PreviousQP(CU(20))dQP(CU(30))=CurrentQP(CU(30))−PreviousQP(CU(21))dQP(CU(31))=CurrentQP(CU(31))−PreviousQP(CU(30))dQP(CU(32))=CurrentQP(CU(32))−PreviousQP(CU(31))dQP(CU(33))=CurrentQP(CU(33))−PreviousQP(CU32))dQP(CU(23))=CurrentQP(CU(23))−PreviousQP(CU33))dQP(CU(3))=CurrentQP(CU(3))−PreviousQP(CU23)

For other LCU(1) to LCU(N) also, the difference values of thequantization parameter are similarly calculated to be transmitted.

In this manner, it is possible to satisfy both of the easiness of thepipeline process and the coding efficiency by adopting advantage of thecharacteristics of each method (indicated by double circle in thedrawing) by calculating and transmitting the difference value of thequantization parameter.

Meanwhile, in view of mounting, when closed control is performed in theLUC, the coding unit CU(0) at the head of the LCU may calculate thedifference value of the quantization parameter by using the equation(15).

Meanwhile, the quantization parameter dQP described above is notrequired to be set for all the coding units, and this may be set onlyfor the CU for which it is desirable to set a value different from areference quantization parameter such as LCUQP, PreviousQP, and SliceQP.

For this purpose, it is also possible to add syntax MinCUForDQPCoded tothe slice header (SliceHeader), for example.

FIG. 26 is a view illustrating an example of the syntax of the sliceheader. The number on a left end of each row is a row number assignedfor description.

In an example in FIG. 26, MinCUForDQPCoded is set in a 22nd line. ThisMinCUForDQPCoded specifies a minimum CU size for which dQP is set. Forexample, even when a minimum size of the CU is 8×8, if it is specifiedthat MinCUForDQPCoded=16, the coding unit quantization parametercalculation unit 334 of the image coding apparatus 300 sets dQP only forthe CU having a size not smaller than 16×16 and does not set dQP for theCU having the size of 8×8. That is, in this case, dQP for the CU havingthe size not smaller than 16×16 is transmitted. Meanwhile,MinCUForDQPCoded may be set as a flag (for example 0:4×4, 1:8×8, 2:16×16and the like) to identify (select) the minimum CU size for which dQP isset from the CU size (4×4, 8×8, 16×16, 32×32 and the like) set at thetime of coding (decoding) as a method of specifying the minimum CU sizefor which dQP is set.

For example, when one who makes an encoder only wants to control withthe CU having the size of 16×16, it is required to transmit all dQP as 0in the CU having the size of 8×8 and this might deteriorate the codingefficiency.

Therefore, by setting such syntax MinCUForDQPCoded, it is possible toomit the transmission of dQP for the CU having the size of 8×8 in thiscase, thereby inhibiting the coding efficiency from deteriorating.

The coding unit quantization value calculation unit 413 of the imagedecoding apparatus 400 grasps that dQP for the CU having the size of 8×8is not transmitted according to such syntax and calculates thequantization value by using the reference quantization parameter such asLCUQP, PreviousQP, and SliceQP.

Meanwhile, MinCUForDQPCoded may be stored in other than the sliceheader. For example, this may be stored in the picture parameter set(PictureParameterSet). It is possible to support operation to changethis value after scene change, for example, by storing the same in theslice header or the picture parameter set.

However, when MinCUForDQPCoded is stored in the slice header, it ispossible to support a case where the picture is multi-sliced andprocessed in parallel for each slice also, which is more desirable.

5. Fifth Embodiment

[Summary]

Although it has been described above that the quantization parameter foreach sub macroblock (coding unit smaller than the LCU) is transmittedfrom the image coding apparatus to the image decoding apparatus, in thiscase, it is required that the image decoding apparatus also may obtainthe quantization parameter for each sub macroblock (coding unit smallerthan the LCU) and perform the quantization for each sub macroblock(coding unit smaller than the LCU) by using the quantization parameter.

Therefore, it may be configured such that the image coding apparatussets the quantization parameter for each macroblock (LCU) and providesthe quantization parameter for each macroblock (LCU) to the imagedecoding apparatus while performing a quantization process for each submacroblock (coding unit smaller than the LCU).

For example, when calculating activity for each macroblock (LCU) by theabove-described TestModel 5, the image coding apparatus calculates theactivity for each block (coding unit) of 8×8, 16×16 and the like,smaller than the macroblock (LCU) even when a size of the macroblock(LCU) is 64×64, 128×128 and the like.

Then, the image coding apparatus determines a quantization parametervalue for each 8×8 block or 16×16 block based on the activity for each8×8 block or 16×16 block based on a method of the TestModel5.

However, the quantization parameter is set for each macroblock (LCU).

For example, suppose that the size of the LCU (macroblock) is 64×64pixels as illustrated in FIG. 27. When the image coding apparatuscalculates the activity for each 16×16 coding unit to calculate thequantization parameter for the LCU, the activity for each coding unit(block) becomes QP₀₀ to QP₃₃.

In a case of the AVC, a quantization parameter QP is designed such thatthe quantization process twice as coarse as the original one isperformed when a value thereof increments by 6 such as from 6 to 12, forexample, as illustrated in FIG. 28.

Deterioration in a chrominance signal is easily noticeable especially ata lower bit rate, that is, with a higher QP. Therefore, a defaultquantization parameter QP_(C) for the chrominance signal is defined inadvance with respect to a quantization parameter QP_(Y) for a luminancesignal.

A user may control this relationship by setting information aboutChromaQPOffset included in image compressed information.

On the other hand, in a case of this embodiment, the image codingapparatus determines a quantization parameter QP_(MB) for the macroblockas represented by a following equation (19) at a first step.

$\begin{matrix}\left\lbrack {{Equation}\mspace{14mu} 3} \right\rbrack & \; \\{{QP}_{MB} = {\min\limits_{{{ij} = 0},3}{QP}_{ij}}} & (19)\end{matrix}$

At a second step, the quantization process for each block is performedby using values of the QP₀₀ to QP₃₃. As a result, a position of anon-zero coefficient in each block is stored in a memory.

At a third step, the quantization process for each block is performed byusing a value of the QP_(MB).

At a fourth step, only a value in the position of the coefficient beingthe non-zero coefficient also at the second step out of the non-zeroobtained at the third step is transmitted to lossless coding informationas coded information.

By performing such a process, although only the QP_(MB) is transmittedto the image compressed information as the quantization parameter, itbecomes possible to realize adaptive quantization and improve subjectiveimage quality of the image compressed information being an output byperforming a pseudo process for each block by using the values of theQP₀₀ to QP₃₃.

[Image Coding Apparatus]

FIG. 29 is a block diagram illustrating a principal configurationexample of the image coding apparatus to which the present technology isapplied. As illustrated in FIG. 29, an image coding apparatus 500 inthis case has a configuration basically similar to that of an imagecoding apparatus 100 in FIG. 1 and performs a similar process.

However, the image coding apparatus 500 includes a rate controller 317,a coding unit quantization unit 504, and a quantization unit 505 inplace of a quantization unit 105, a rate controller 117, and a submacroblock quantization unit 121 of the image coding apparatus 100.

Although the image coding apparatus 100 in FIG. 1 includes a submacroblock inverse quantization unit 122 in addition to an inversequantization unit 108, the image coding apparatus 500 only includes theinverse quantization unit 108. That is, an inverse quantization processis performed for each LCU (macroblock) as in conventional AVC and thelike. This also applies to the image decoding apparatus corresponding tothe image coding apparatus 500.

The coding unit quantization unit 504 performs the quantization for eachcoding unit (for example, SCU) by using the activity for each codingunit obtained by the rate controller 317.

The quantization unit 505 obtains the quantization parameter for eachLCU and performs the quantization for each coding unit by using thesame. Then, the quantization unit 505 replaces the non-zero coefficientout of quantized orthogonal transform coefficients of the coding unitsobtained by the coding unit quantization unit 504 with a result of thequantization process by the quantization unit 505 (quantized orthogonaltransform coefficient) in the same position.

A result of this replacement is supplied to the lossless coding unit 106and the inverse quantization unit 108 as a result of the quantization.Also, the quantization parameter for each LCU calculated by thequantization unit 505 is supplied to the lossless coding unit 106 andthe inverse quantization unit 108.

The inverse quantization unit 108 and an inverse quantization unit ofthe image decoding apparatus (not illustrated) perform inversequantization by using the quantization parameter for each LCU as in thecase of the conventional AVC and the like.

[Configurations of Rate Controller, Coding Unit Quantization Unit, andQuantization Unit]

FIG. 30 is a block diagram illustrating a detailed configuration exampleof the rate controller, the coding unit quantization unit, and thequantization unit in FIG. 29.

As illustrated in FIG. 30, the coding unit quantization unit 504includes a coding unit quantization parameter determination unit 511, acoding unit quantization processor 512, and a non-zero coefficientposition buffer 513.

The coding unit quantization parameter determination unit 511 determinesa quantization parameter CU_QP for each coding unit (for example, SCU)in a layer lower than the LCU by using the activity for each coding unit(for example, SCU) in the layer lower than the LCU supplied from anactivity buffer 322 of the rate controller 317. The coding unitquantization parameter determination unit 511 supplies the quantizationparameter CU_QP for each coding unit to the coding unit quantizationprocessor 512 and an LCU quantization parameter determination unit 522of the quantization unit 505.

The coding unit quantization processor 512 quantizes the orthogonaltransform coefficient supplied from an orthogonal transform coefficientbuffer 521 of the quantization unit 505 for each coding unit (forexample, SCU) in the layer lower than the LCU by using the quantizationparameter CU_QP for each coding unit supplied from the coding unitquantization parameter determination unit 511. The coding unitquantization processor 512 supplies the position of the coding unit inwhich the value is not 0 (non-zero coefficient) out of the quantizedorthogonal transform coefficients of the coding units obtained by thequantization to the non-zero coefficient position buffer 513 and allowthe same to hold this position.

The non-zero coefficient position buffer 513 supplies the held positionof the non-zero coefficient to a coefficient replacing unit 524 of thequantization unit 505 at a predetermined timing.

As illustrated in FIG. 30, the quantization unit 505 includes theorthogonal transform coefficient buffer 521, the LCU quantizationparameter determination unit 522, an LCU quantization processor 523, andthe coefficient replacing unit 524.

The orthogonal transform coefficient buffer 521 holds the orthogonaltransform coefficient supplied from an orthogonal transformation unit104 and supplies the held orthogonal transform coefficient to the codingunit quantization processor 512 and the LCU quantization processor 523at a predetermined timing.

The LCU quantization parameter determination unit 522 determines aminimum value in the LCU of the quantization parameters CU_QP for eachcoding unit supplied from the coding unit quantization parameterdetermination unit 511 as a quantization parameter LCU_QP for each LCUas represented by the above-described equation (19). The LCUquantization parameter determination unit 522 supplies the quantizationparameter LCU_QP (minimum value of CU_QP in the current LCU) for eachLCU to the LCU quantization processor 523.

The LCU quantization processor 523 quantizes the orthogonal transformcoefficient supplied from the orthogonal transform coefficient buffer521 for each coding unit (for example, SCU) in the layer lower than theLCU by using the quantization parameter LCU_QP for each LCU suppliedfrom the LCU quantization parameter determination unit 522. The LCUquantization processor 523 supplies the quantized orthogonal transformcoefficient for each coding unit obtained by the quantization to thecoefficient replacing unit 524.

The coefficient replacing unit 524 replaces the coefficient in theposition different from the position of the non-zero coefficientsupplied from the non-zero coefficient position buffer 513 out of thecoefficients of which value is not 0 (non-zero coefficient) of theorthogonal transform coefficients quantized by the LCU quantizationprocessor 523 with 0.

That is, the coefficient replacing unit 524 adopts a value of the resultof the quantization as the quantized orthogonal transform coefficientonly for the coding unit (in the layer lower than the LCU) in which theobtained value of the result of the quantization is not 0 in both of thequantization using the quantization parameter CU_QP determined for eachcoding unit in the layer lower than the LCU and the quantization usingthe quantization parameter LCU_QP determined for each LCU. On the otherhand, the coefficient replacing unit 524 sets all the values of all ofthe quantized orthogonal transform coefficients to 0 for other codingunits (in the layer lower than the LCU).

The coefficient replacing unit 524 supplies the quantized orthogonaltransform coefficient of which value is appropriately replaced in thismanner to the lossless coding unit 106 and the inverse quantization unit108 together with the quantization parameter LCU_QP determined for eachLCU.

The lossless coding unit 106 codes supplied coefficient data andquantization parameter to supply to the image decoding apparatus(capable of decoding coded data generated by the image coding apparatus500) corresponding to the image coding apparatus 500. The image decodingapparatus performs the inverse quantization by using the quantizationparameter LCU_QP for each LCU supplied from the image coding apparatus500 as in the case of the conventional AVC and the like.

The inverse quantization unit 108 similarly inversely quantizes thecoefficient data supplied from the coefficient replacing unit 524 byusing the quantization parameter LCU_QP for each LCU supplied from thecoefficient replacing unit 524.

Meanwhile, the inverse quantization unit 108 has a configurationbasically similar to that of the inverse quantization unit 203 describedwith reference to FIG. 10. However, in the case of the inversequantization unit 108, an inverse quantization processor 253 inverselyquantizes the quantized orthogonal transform coefficient supplied froman orthogonal transform coefficient buffer 252 by using the quantizationparameter (quantization parameter LCU_QP for each LCU) supplied from aquantization parameter buffer 251.

[Flow of Coding Process]

Next, an example of a flow of a coding process executed by the imagecoding apparatus 500 is described with reference to a flowchart in FIG.31. In this case, each process of the coding process is performed in amanner basically similar to that in each process of the coding processdescribed with reference to the flowchart in FIG. 7.

That is, processes at steps S701 to S704 are performed as the processesat steps S101 to S104 in FIG. 7. However, the quantization process atstep S705 is performed in place of steps S105 and S106 in FIG. 7. Also,processes at steps S706 to S716 are performed as the processes at stepsS106 to S117.

[Flow of Quantization Process]

Next, an example of a flow of the quantization process executed at stepS705 in FIG. 31 is described with reference to a flowchart in FIG. 32.

When the quantization process is started, an activity calculation unit321 calculates the activity for each coding unit at step S731.

At step S732, the coding unit quantization parameter determination unit511 determines the quantization parameter CU_QP for each coding unit inthe layer lower than the LCU.

At step S733, the LCU quantization parameter determination unit 522determines the quantization parameter LCU_QP for each LCU.

At step S734, the coding unit quantization processor 512 performs thequantization by using the quantization parameter CU_QP for each codingunit in the layer lower than the LCU.

At step S735, the non-zero coefficient position buffer 513 holds theposition of the non-zero coefficient generated by the quantizationprocess at step S734.

At step S736, the LCU quantization processor 523 performs thequantization by using the quantization parameter LCU_QP for each LCU.

At step S737, the coefficient replacing unit 524 replaces the value ofthe quantized orthogonal transform coefficient of the coding unit in thelayer lower than the LCU in the position different from the position ofthe non-zero coefficient held by the process at step S735 with 0.

When replacement is finished, the quantization process is finished andthe process is returned to step S705 in FIG. 31, then the processes atstep S706 and subsequent steps are executed.

As described above, in an image information coding apparatus and animage information decoding apparatus of which output and input are theimage compressed information, respectively, based on a coding systemusing an extended macroblock, it is possible to perform the adaptivequantization based on characteristics of a flat area and a texture areaeven when they are mixed in a single LCU (macroblock) by performing thepseudo quantization process for each code unit (sub macroblock) in thelayer lower than the LCU, thereby improving the subjective imagequality.

6. Sixth Embodiment

[Application to Multi-View Image Coding/Multi-View Image Decoding]

The above-described series of processes may be applied to multi-viewimage coding and multi-view image decoding. FIG. 33 illustrates anexample of a multi-view image coding system.

As illustrated in FIG. 33, a multi-view image includes images from aplurality of viewpoints and an image from a predetermined viewpoint outof a plurality of viewpoints is specified as a base view image. An imagefrom each viewpoint other than the base view image is treated asnon-base view image.

When the multi-view image coding as illustrated in FIG. 33 is performed,it is also possible to obtain difference between quantization parametersof each view (identical view).

(1) Base-View:dQP(base view)=Current_CU_QP(base view)−LCU_QP(base view)  (1-1)dQP(base view)=Current_CU_QP(base view)−Previous_CU_QP(base view)  (1-2)dQP(base view)=Current_CU_QP(base view)−Slice_QP(base view)  (1-3)(2) Non-Base-View:dQP(non-base view)=Current_CU_QP(non-base view)−LCU_QP(non-baseview)  (2-1)dQP(non-base view)=CurrentQP(non-base view)−PrevioussQP(non-baseview)  (2-2)dQP(non-base view)=Current_CU_QP(non-base view)−Slice_QP(non-baseview)  (2-3)

When the multi-view image coding is performed, it is also possible toobtain the difference between the quantization parameters for each view(different views).

(3) Base-View/Non-Base View:dQP(inter-view)=Slice_QP(base view)−Slice_QP(non-base view)  (3-1)dQP(inter-view)=LCU_QP(base view)−LCU_QP(non-base view)  (3-2)(4) non-base view/non-base view:dQP(inter-view)=Slice_QP(non-base view i)−Slice_QP(non-base viewj)  (4-1)dQP(inter-view)=LCU_QP(non-base view i)−LCU_QP(non-base view j)  (4-2)

In this case, it is also possible to combine the above-described (1) to(4). For example, in the non-base view, a method of obtaining thedifference between the quantization parameters in a slice level betweenthe base view and the non-base view (3-1 and 2-3 are combined) and amethod of obtaining the difference between the quantization parametersin an LCU level between the base view and the non-base view (3-2 and 2-1are combined) are considered. In this manner, it is possible to improvecoding efficiency also in a case where the multi-view coding isperformed by repeatedly applying the difference.

As in the above-described method, it is also possible to set a flag toidentify whether there is dQP of which value is not 0 for each dQPdescribed above.

[Multi-View Image Coding Apparatus]

FIG. 34 is a view illustrating a multi-view image coding apparatus,which performs the above-described multi-view image coding. Asillustrated in FIG. 34, a multi-view image coding apparatus 600 includesa coding unit 601, a coding unit 602, and a multiplexing unit 603.

The coding unit 601 codes the base view image to generate a base viewimage coded stream. The coding unit 602 codes the non-base view image togenerate a non-base view image coded stream. The multiplexing unit 603multiplexes the base view image coded stream generated by the codingunit 601 and the non-base view image coded stream generated by thecoding unit 602 to generate a multi-view image coded stream.

An image coding apparatus 100 (FIG. 1), an image coding apparatus 300(FIG. 18), or an image coding apparatus 500 (FIG. 29) may be applied tothe coding unit 601 and the coding unit 602 of the multi-view imagecoding apparatus 600. In this case, the multi-view image codingapparatus 600 sets a difference value between the quantization parameterset by the coding unit 601 and the quantization parameter set by thecoding unit 602 to transmit.

[Multi-View Image Decoding Apparatus]

FIG. 35 is a view illustrating a multi-view image decoding apparatus,which performs the above-described multi-view image decoding. Asillustrated in FIG. 35, a multi-view image decoding apparatus 610includes a demultiplexing unit 611, a decoding unit 612, and a decodingunit 613.

The demultiplexing unit 611 demultiplexes the multi-view image codedstream in which the base view image coded stream and the non-base viewimage coded stream are multiplexed to extract the base view image codedstream and the non-base view image coded stream. The decoding unit 612decodes the base view image coded stream extracted by the demultiplexingunit 611 to obtain the base view image. The decoding unit 613 decodesthe non-base view image coded stream extracted by the demultiplexingunit 611 to obtain the non-base view image.

It is possible to apply an image decoding apparatus 200 (FIG. 9) or animage decoding apparatus 400 (FIG. 21) to the decoding unit 612 and thedecoding unit 613 of the multi-view image decoding apparatus 610. Inthis case, the multi-view image decoding apparatus 610 sets thequantization parameter from the difference value between thequantization parameter set by the coding unit 601 and the quantizationparameter set by the coding unit 602 to perform inverse quantization.

7. Seventh Embodiment

[Application to Hierarchical Image Point Coding/Hierarchical ImageDecoding]

The above-described series of processes may be applied to hierarchicalimage coding/hierarchical image decoding. FIG. 36 illustrates an exampleof a multi-view image coding system.

As illustrated in FIG. 36, a hierarchical image includes images of aplurality of layers (resolutions) and an image of a predetermined layerout of a plurality of resolutions is specified as a base layer image. Animage of each layer other than the base layer image is treated as anon-base layer image.

When the hierarchical image coding (spatial scalability) as illustratedin FIG. 36 is performed, it is also possible to obtain differencebetween quantization parameters of each layer (identical layer):

(1) Base-Layer:dQP(base layer)=Current_CU_QP(base layer)−LCU_QP(base layer)  (1-1)dQP(base layer)=Current_CU_QP(base layer)−Previous_CU_QP(baselayer)  (1-2)dQP(base layer)=Current_CU_QP(base layer)−Slice_QP(base layer)  (1-3)(2) Non-Base-Layer:dQP(non-base layer)=Current_CU_QP(non-base layer)−LCU_QP(non-baselayer)  (2-1)dQP(non-base layer)=CurrentQP(non-base layer)−PrevioussQP(non-baselayer)  (2-2)dQP(non-base layer)=Current_CU_QP(non-base layer)−Slice_QP(non-baselayer)  (2-3)

When the hierarchical coding is performed, it is also possible to obtainthe difference between the quantization parameters of each layer(different layers).

(3) Base-Layer/Non-Base Layer:dQP(inter-layer)=Slice_QP(base layer)−Slice_QP(non-base layer)  (3-1)dQP(inter-layer)=LCU_QP(base layer)−LCU_QP(non-base layer)  (3-2)(4) Non-Base Layer/Non-Base Layer:dQP(inter-layer)=Slice_QP(non-base layer i)−Slice_QP(non-base layerj)  (4-1)dQP(inter-layer)=LCU_QP(non-base layer i)−LCU_QP(non-base layerj)  (4-2)

In this case, it is also possible to combine the above-described (1) to(4). For example, in the non-base layer, a method of obtaining thedifference between the quantization parameters in a slice level betweenthe base layer and the non-base layer (3-1 and 2-3 are combined) and amethod of obtaining the difference between the quantization parametersin an LCU level between the base layer and the non-base layer (3-2 and2-1 are combined) are considered. In this manner, by repeatedly applyingthe difference, it becomes possible to improve coding efficiency alsowhen the hierarchical coding is performed.

It is also possible to set a flag to identify whether there is dQP ofwhich value is not 0 for each dQP described above as in theabove-described method.

[Hierarchical Image Coding Apparatus]

FIG. 37 is a view illustrating a hierarchical image coding apparatus,which performs the above-described hierarchical image coding. Asillustrated in FIG. 37, a hierarchical image coding apparatus 620includes a coding unit 621, a coding unit 622, and a multiplexing unit623.

The coding unit 621 codes the base layer image to generate a base layerimage coded stream. The coding unit 622 codes the non-base layer imageto generate a non-base layer image coded stream. The multiplexing unit623 multiplexes the base layer image coded stream generated by thecoding unit 621 and the non-base layer image coded stream generated bythe coding unit 622 to generate a hierarchical image coded stream.

An image coding apparatus 100 (FIG. 1), an image coding apparatus 300(FIG. 18), or an image coding apparatus 500 (FIG. 29) may be applied tothe coding unit 621 and the coding unit 622 of the hierarchical imagecoding apparatus 620. In this case, the hierarchical image codingapparatus 600 sets a difference value between the quantization parameterset by the coding unit 621 and the quantization parameter set by thecoding unit 622 to transmit.

[Hierarchical Image Decoding Apparatus]

FIG. 38 is a view illustrating a hierarchical image decoding apparatus,which performs the above-described hierarchical image decoding. Asillustrated in FIG. 38, a hierarchical image decoding apparatus 630includes a demultiplexing unit 631, a decoding unit 632, and a decodingunit 633.

The demultiplexing unit 631 demultiplexes the hierarchical image codedstream obtained by multiplexing the base layer image coded stream andthe non-base layer image coded stream to extract the base layer imagecoded stream and the non-base layer image coded stream. The decodingunit 632 decodes the base layer image coded stream extracted by thedemultiplexing unit 631 to obtain the base layer image. The decodingunit 633 decodes the non-base layer image coded stream extracted by thedemultiplexing unit 631 to obtain the non-base layer image.

An image decoding apparatus 200 (FIG. 9) or an image decoding apparatus400 (FIG. 21) may be applied to the decoding unit 632 and the decodingunit 633 of the hierarchical image decoding apparatus 630. In this case,the hierarchical image decoding apparatus 630 sets the quantizationparameter from the difference value between the quantization parameterset by the coding unit 631 and the quantization parameter set by thecoding unit 632 to perform inverse quantization.

8. Eighth Embodiment

[Computer]

It is possible that the above-described series of processes is executedby hardware or executed by software. In this case, it may be configuredas a computer illustrated in FIG. 39, for example.

In FIG. 39, a CPU (Central Processing Unit) 701 of a personal computer700 executes various processes according to a program stored in a ROM(Read Only Memory) 702 or a program loaded from a storage unit 713 intoa RAM (Random Access Memory) 703. Data necessary for the CPU 701 toexecute the various processes also are appropriately stored in the RAM703.

The CPU 701, the ROM 702, and the RAM 703 are connected to one anotherthrough a bus 704. An input/output interface 710 also is connected tothe bus 704.

An input unit 711 formed of a keyboard, a mouse and the like, an outputunit 712 formed of a display formed of a CRT (Cathode Ray Tube) and anLCD (Liquid Crystal Display), a speaker and the like, the storage unit713 formed of a hard disc and the like, and a communication unit 714formed of a modem and the like are connected to the input/outputinterface 710. The communication unit 714 performs a communicationprocess through a network including the Internet.

A drive 715 is connected to the input/output interface 710 as needed, aremovable medium 721 such as a magnetic disc, an optical disc, amagnetooptical disc, and a semiconductor memory is appropriately mountedthereon, and a computer program read from the medium is installed on thestorage unit 713 as needed.

When the above-described series of processes is executed by thesoftware, a program, which composes the software, is installed from thenetwork or a recording medium.

The recording medium is composed not only of the removable medium 721including the magnetic disc (including a flexible disc), the opticaldisc (including a CD-ROM (Compact Disc-Read Only Memory) and a DVD(Digital Versatile Disc)), the magnetooptical disc (including an MD(Mini Disc)), and the semiconductor memory, in which the program isrecorded, distributed to a user for distributing the program separatelyfrom an apparatus main body but also of the ROM 702 in which the programis recorded and the hard disc included in the storage unit 713distributed to the user in a state being embedded in advance in theapparatus main body as illustrated in FIG. 39, for example.

Meanwhile, the program executed by the computer may be the program ofwhich processes are chronologically performed in order described in thisspecification or the program of which processes are performed inparallel or at a required timing such as when this is called.

Also, in this specification, a step of describing the program recordedin the recording medium includes not only the processes chronologicallyperformed in the described order but also the processes executed inparallel or individually, which are not necessarily chronologicallyperformed.

Also, in this specification, a system means an entire apparatusincluding a plurality of devices (apparatus).

It is also possible to divide the configuration described above as oneapparatus (or processor) into a plurality of apparatuses (orprocessors). Other way round, it is also possible to put theconfigurations described above as a plurality of apparatuses (orprocessors) together as one apparatus (or processor). It goes withoutsaying that a configuration other than the above-described one may beadded to the configuration of each apparatus (or each processor).Further, it is also possible to add a part of the configuration of acertain apparatus (or processor) to the configuration of anotherapparatus (or another processor) as long as the configuration andoperation as an entire system are substantially the same. That is, anembodiment of the present technology is not limited to theabove-described embodiments and various modifications may be madewithout departing from the spirit of the present technology.

An image coding apparatus 100 (FIG. 1), an image coding apparatus 300(FIG. 18), an image coding apparatus 500 (FIG. 29), a multi-view imagecoding apparatus 600 (FIG. 34), a hierarchical image coding apparatus620 (FIG. 37), an image decoding apparatus 200 (FIG. 9), an imagedecoding apparatus 400 (FIG. 21), a multi-view image decoding apparatus610 (FIG. 35), and a hierarchical image decoding apparatus 630 (FIG. 38)according to the above-described embodiments are applicable to variouselectronic devices such as a transmitter or a receiver in satellitebroadcasting, cable broadcasting to a cable television and the like,distribution on the Internet, distribution to a terminal throughcellular communication and the like, a recording apparatus, whichrecords an image on the medium such as the optical disc, the magneticdisc, and a flash memory, or a reproducing apparatus, which reproducesthe image from a storage medium. Four applications are hereinafterdescribed.

[Television Apparatus]

FIG. 40 illustrates an example of a schematic configuration of atelevision apparatus to which the above-described embodiment is applied.A television apparatus 900 includes an antenna 901, a tuner 902, ademultiplexer 903, a decoder 904, a video signal processor 905, adisplay unit 906, a voice signal processor 907, a speaker 908, anexternal interface 909, a controller 910, a user interface 911, and abus 912.

The tuner 902 extracts a signal of a desired channel from a broadcastsignal received through the antenna 901 and demodulates the extractedsignal. Then, the tuner 902 outputs a coded bit stream obtained bydemodulation to the demultiplexer 903. That is, the tuner 902 serves astransmitting means in the television apparatus 900, which receives thecoded stream in which the image is coded.

The demultiplexer 903 separates a video stream and a voice stream of aprogram to be watched from the coded bit stream and outputs eachseparated stream to the decoder 904.

Also, the demultiplexer 903 extracts auxiliary data such as EPG(Electronic Program Guide) from the coded bit stream and supplies theextracted data to the controller 910. Meanwhile, the demultiplexer 903may descramble when the coded bit stream is scrambled.

The decoder 904 decodes the video stream and the voice stream input fromthe demultiplexer 903. Then, the decoder 904 outputs video datagenerated by a decoding process to the video signal processor 905. Also,the decoder 904 outputs voice data generated by the decoding process tothe voice signal processor 907.

The video signal processor 905 reproduces the video data input from thedecoder 904 and allows the display unit 906 to display video. The videosignal processor 905 may also allow the display unit 906 to display anapplication screen supplied through the network. The video signalprocessor 905 may also perform an additional process such as noiseremoval, for example, to the video data according to setting. Further,the video signal processor 905 may generate a GUI (Graphical UserInterface) image such as a menu, a button, and a cursor, for example,and superimpose the generated image on an output image.

The display unit 906 is driven by a drive signal supplied from the videosignal processor 905 to display the video or image on a video screen ofa display device (for example, a liquid crystal display, a plasmadisplay, an OELD (Organic ElectroLuminescence Display (organic ELdisplay) and the like).

The voice signal processor 907 performs a reproducing process such asD/A conversion and amplification to the voice data input from thedecoder 904 and allows the speaker 908 to output the voice. The voicesignal processor 907 may also perform an additional process such as thenoise removal to the voice data.

The external interface 909 is the interface for connecting thetelevision apparatus 900 and an external device or the network. Forexample, the video stream or the voice stream received through theexternal interface 909 may be decoded by the decoder 904. That is, theexternal interface 909 also serves as the transmitting means in thetelevision apparatus 900, which receives the coded stream in which theimage is coded.

The controller 910 includes a processor such as the CPU and a memorysuch as the RAM and the ROM. The memory stores the program executed bythe CPU, program data, the EPG data, data obtained through the networkand the like. The program stored in the memory is read by the CPU atstartup of the television apparatus 900 to be executed, for example. TheCPU controls operation of the television apparatus 900 according to anoperation signal input from the user interface 911, for example, byexecuting the program.

The user interface 911 is connected to the controller 910. The userinterface 911 includes a button and a switch for the user to operate thetelevision apparatus 900, a receiver of a remote control signal and thelike, for example. The user interface 911 detects operation by the userthrough the components to generate the operation signal and outputs thegenerated operation signal to the controller 910.

The bus 912 connects the tuner 902, the demultiplexer 903, the decoder904, the video signal processor 905, the voice signal processor 907, theexternal interface 909, and the controller 910 to one another.

In the television apparatus 900 configured in this manner, the decoder904 has functions of the image decoding apparatus 200 (FIG. 9), theimage decoding apparatus 400 (FIG. 21), the multi-view image decodingapparatus 610 (FIG. 35), or the hierarchical image decoding apparatus630 (FIG. 38) according to the above-described embodiments. Therefore,the decoder 904 calculates a quantization value for each sub macroblockby using a quantization parameter such as submb_qp_delta supplied from acoding side to perform inverse quantization for the video decoded by thetelevision apparatus 900. Therefore, it is possible to perform aninverse quantization process more suitable for contents of the image,thereby inhibiting subjective image quality of a decoded image fromdeteriorating.

[Mobile Phone]

FIG. 41 illustrates an example of a schematic configuration of a mobilephone to which the above-described embodiment is applied. A mobile phone920 is provided with an antenna 921, a communication unit 922, a voicecodec 923, a speaker 924, a microphone 925, a camera unit 926, an imageprocessor 927, a multiplexing/separating unit 928, arecording/reproducing unit 929, a display unit 930, a controller 931, anoperation unit 932, and a bus 933.

The antenna 921 is connected to the communication unit 922. The speaker924 and the microphone 925 are connected to the voice codec 923. Theoperation unit 932 is connected to the controller 931. The bus 933connects the communication unit 922, the voice codec 923, the cameraunit 926, the image processor 927, the multiplexing/separating unit 928,the recording/reproducing unit 929, the display unit 930, and thecontroller 931 to one another.

The mobile phone 920 performs operation such as transmission/receptionof a voice signal, transmission/reception of an e-mail or image data,image taking, and recording of data in various operation modes includinga voice communication mode, a data communication mode, an imaging mode,and a television-phone mode.

In the voice communication mode, an analog voice signal generated by themicrophone 925 is supplied to the voice codec 923. The voice codec 923converts the analog voice signal to the voice data and A/D converts theconverted voice data to compress. Then, the voice codec 923 outputs thecompressed voice data to the communication unit 922. The communicationunit 922 codes and modulates the voice data to generate a transmissionsignal. Then, the communication unit 922 transmits the generatedtransmission signal to a base station (not illustrated) through theantenna 921. Also, the communication unit 922 amplifies a wirelesssignal received through the antenna 921 and applies frequency conversionto the same to obtain a reception signal. Then, the communication unit922 generates the voice data by demodulating and decoding the receptionsignal and outputs the generated voice data to the voice codec 923. Thevoice codec 923 expands the voice data and D/A converts the same togenerate the analog voice signal. Then, the voice codec 923 supplies thegenerated voice signal to the speaker 924 to allow the same to outputthe voice.

In the data communication mode, for example, the controller 931generates character data composing the e-mail according to the operationby the user through the operation unit 932. Also, the controller 931allows the display unit 930 to display characters. The controller 931generates e-mail data according to a transmission instruction from theuser through the operation unit 932 to output the generated e-mail datato the communication unit 922. The communication unit 922 codes andmodulates the e-mail data to generate the transmission signal. Then, thecommunication unit 922 transmits the generated transmission signal tothe base station (not illustrated) through the antenna 921. Also, thecommunication unit 922 amplifies the wireless signal received throughthe antenna 921 and applies the frequency conversion to the same toobtain the reception signal. Then, the communication unit 922demodulates and decodes the reception signal to restore the e-mail dataand outputs the restored e-mail data to the controller 931. Thecontroller 931 allows the display unit 930 to display contents of thee-mail data and allows the storage medium of the recording/reproducingunit 929 to store the e-mail data.

The recording/reproducing unit 929 includes an arbitraryreadable/writable storage medium. For example, the storage medium may bea built-in storage medium such as the RAM and the flash memory and maybe an externally-mounted storage medium such as the hard disc, themagnetic disc, the magnetooptical disc, the optical disc, a USB memory,and a memory card.

In the imaging mode, for example, the camera unit 926 takes an image ofan object to generate the image data and outputs the generated imagedata to the image processor 927. The image processor 927 codes the imagedata input from the camera unit 926 and stores the coded stream in thestorage medium of the recording/reproducing unit 929.

Also, in the television-phone mode, for example, themultiplexing/separating unit 928 multiplexes the video stream coded bythe image processor 927 and the voice stream input from the voice codec923 and outputs the multiplexed stream to the communication unit 922.The communication unit 922 codes and modulates the stream to generatethe transmission signal. Then, the communication unit 922 transmits thegenerated transmission signal to the base station (not illustrated)through the antenna 921. Also, the communication unit 922 amplifies thewireless signal received through the antenna 921 and applies thefrequency conversion to the same to obtain the reception signal. Thetransmission signal and the reception signal may include the coded bitstream. Then, the communication unit 922 restores the stream bydemodulating and decoding the reception signal and outputs the restoredstream to the multiplexing/separating unit 928. Themultiplexing/separating unit 928 separates the video stream and thevoice stream from the input stream and outputs the video stream and thevoice stream to the image processor 927 and the voice codec 923,respectively. The image processor 927 decodes the video stream togenerate the video data. The video data is supplied to the display unit930 and a series of images is displayed by the display unit 930. Thevoice codec 923 expands the voice stream and D/A converts the same togenerate the analog voice signal. Then, the voice codec 923 supplies thegenerated voice signal to the speaker 924 to output the voice.

In the mobile phone 920 configured in this manner, the image processor927 has the functions of the image coding apparatus 100 (FIG. 1), theimage coding apparatus 300 (FIG. 18), the image coding apparatus 500(FIG. 29), the multi-view image coding apparatus 600 (FIG. 34), or thehierarchical image coding apparatus 620 (FIG. 37) and the functions ofthe image decoding apparatus 200 (FIG. 9), the image decoding apparatus400 (FIG. 21), the multi-view image decoding apparatus 610 (FIG. 35), orthe hierarchical image decoding apparatus 630 (FIG. 38) according to theabove-described embodiments. Therefore, the image processor 927calculates the quantization value for each sub macroblock and quantizesan orthogonal transform coefficient by using the quantization value foreach sub macroblock for the video coded and decoded by the mobile phone920. In this manner, it is possible to perform the quantization processmore suitable for the contents of the image and generate the coded dataso as to inhibit the subjective image quality of the decoded image fromdeteriorating. Also, the image processor 927 calculates the quantizationvalue for each sub macroblock by using the quantization parameter suchas submb_qp_delta supplied from the coding side to perform the inversequantization. Therefore, it is possible to perform the inversequantization process more suitable for the contents of the image toinhibit the subjective image quality of the decoded image fromdeteriorating.

Although it has been described above as the mobile phone 920, the imagecoding apparatus and the image decoding apparatus to which the presenttechnology is applied may be applied to any apparatus having an imagingfunction and a communication function similar to those of the mobilephone 920 such as a PDA (Personal Digital Assistants), a smartphone, aUMPC (Ultra Mobile Personal Computer), a netbook, and a notebookcomputer, for example, as in the case of the mobile phone 920.

[Recording/Reproducing Apparatus]

FIG. 42 illustrates an example of a schematic configuration of therecording/reproducing apparatus to which the above-described embodimentis applied. The recording/reproducing apparatus 940 codes the voice dataand the video data of a received broadcast program to record on therecording medium, for example. Also, the recording/reproducing apparatus940 may code the voice data and the video data obtained from anotherapparatus to record on the recording medium, for example. Also, therecording/reproducing apparatus 940 reproduces the data recorded on therecording medium by a monitor and the speaker according to theinstruction of the user. At that time, the recording/reproducingapparatus 940 decodes the voice data and the video data.

The recording/reproducing apparatus 940 is provided with a tuner 941, anexternal interface 942, an encoder 943, a HDD (Hard Disk Drive) 944, adisc drive 945, a selector 946, a decoder 947, an OSD (On-ScreenDisplay) 948, a controller 949, and a user interface 950.

The tuner 941 extracts a signal of a desired channel from the broadcastsignal received through an antenna (not illustrated) and demodulates theextracted signal. Then, the tuner 941 outputs the coded bit streamobtained by the demodulation to the selector 946. That is, the tuner 941serves as the transmitting means in the recording/reproducing apparatus940.

The external interface 942 is the interface for connecting therecording/reproducing apparatus 940 and the external device or thenetwork. The external interface 942 may be an IEEE1394 interface, anetwork interface, a USB interface, a flash memory interface and thelike, for example. For example, the video data and the voice datareceived through the external interface 942 are input to the encoder943. That is, the external interface 942 serves as the transmittingmeans in the recording/reproducing apparatus 940.

The encoder 943 codes the video data and the voice data when the videodata and the voice data input from the external interface 942 are notcoded. Then, the encoder 943 outputs the coded bit stream to theselector 946.

The HDD 944 records the coded bit stream in which content data such asthe video and the voice are compressed, various programs and other dataon an internal hard disc. The HDD 944 reads the data from the hard discwhen reproducing the video and the voice.

The disc drive 945 records and reads the data on and from the mountedrecording medium. The recording medium mounted on the disc drive 945 maybe the DVD disc (DVD-Video, DVD-RAM, DVD-R, DVD-RW, DVD+R, DVD+RW andthe like), a Blu-ray (registered trademark) disc and the like, forexample.

The selector 946 selects the coded bit stream input from the tuner 941or the encoder 943 and outputs the selected coded bit stream to the HDD944 or the disc drive 945 when recording the video and the voice. Also,the selector 946 outputs the coded bit stream input from the HDD 944 orthe disc drive 945 to the decoder 947 when reproducing the video and thevoice.

The decoder 947 decodes the coded bit stream to generate the video dataand the voice data. Then, the decoder 947 outputs the generated videodata to the OSD 948. Also, the decoder 904 outputs the generated voicedata to an external speaker.

The OSD 948 reproduces the video data input from the decoder 947 todisplay the video. The OSD 948 may also superimpose the GUI image suchas the menu, the button, and the cursor, for example, on the displayedvideo.

The controller 949 includes the processor such as the CPU and the memorysuch as the RAM and ROM. The memory stores the program executed by theCPU, the program data and the like. The program stored in the memory isread by the CPU to be executed on activation of therecording/reproducing apparatus 940, for example. The CPU controlsoperation of the recording/reproducing apparatus 940 according to anoperation signal input from the user interface 950, for example, byexecuting the program.

The user interface 950 is connected to the controller 949. The userinterface 950 includes a button and a switch for the user to operate therecording/reproducing apparatus 940 and a receiver of a remote controlsignal, for example. The user interface 950 detects operation by theuser through the components to generate the operation signal and outputsthe generated operation signal to the controller 949.

In the recording/reproducing apparatus 940 configured in this manner,the encoder 943 has the functions of the image coding apparatus 100(FIG. 1), the image coding apparatus 300 (FIG. 18), the image codingapparatus 500 (FIG. 29), the multi-view image coding apparatus 600 (FIG.34), or the hierarchical image coding apparatus 620 (FIG. 37) accordingto the above-described embodiments. Also, the decoder 947 has thefunctions of the image decoding apparatus 200 (FIG. 9), the imagedecoding apparatus 400 (FIG. 21), the multi-view image decodingapparatus 610 (FIG. 35), and the hierarchical image decoding apparatus630 (FIG. 38) according to the above-described embodiments. Therefore,the quantization value for each sub macroblock is calculated and theorthogonal transform coefficient is quantized by using the quantizationvalue for each sub macroblock for the video coded and decoded by therecording/reproducing apparatus 940. In this manner, it is possible toperform the quantization process more suitable for the contents of theimage and generate the coded data so as to inhibit the subjective imagequality of the decoded image from deteriorating. Also, the quantizationvalue for each sub macroblock is calculated by using the quantizationparameter such as submb_qp_delta supplied from the coding side and theinverse quantization is performed. Therefore, it is possible to performthe inverse quantization process more suitable for the contents of theimage and inhibit the subjective image quality of the decoded image fromdeteriorating.

[Imaging Apparatus]

FIG. 43 illustrates an example of a schematic configuration of animaging apparatus to which the above-described embodiment is applied. Animaging apparatus 960 takes an image of the object to generate theimage, codes the image data and records the same on the recordingmedium.

The imaging apparatus 960 is provided with an optical block 961, animaging unit 962, a signal processor 963, an image processor 964, adisplay unit 965, an external interface 966, a memory 967, a media drive968, an OSD 969, a controller 970, a user interface 971, and a bus 972.

The optical block 961 is connected to the imaging unit 962. The imagingunit 962 is connected to the signal processor 963. The display unit 965is connected to the image processor 964. The user interface 971 isconnected to the controller 970. The bus 972 connects the imageprocessor 964, the external interface 966, the memory 967, the mediadrive 968, the OSD 969, and the controller 970 to one another.

The optical block 961 includes a focus lens, a diaphragm mechanism andthe like. The optical block 961 forms an optical image of the object onan imaging surface of the imaging unit 962. The imaging unit 962includes an image sensor such as a CCD and a CMOS and converts theoptical image formed on the imaging surface to an image signal as anelectric signal by photoelectric conversion. Then, the imaging unit 962outputs the image signal to the signal processor 963.

The signal processor 963 performs various camera signal processes suchas knee correction, gamma correction, color correction to the imagesignal input from the imaging unit 962. The signal processor 963 outputsthe image data after the camera signal process to the image processor964.

The image processor 964 codes the image data input from the signalprocessor 963 to generate the coded data. Then, the image processor 964outputs the generated coded data to the external interface 966 or themedia drive 968. Also, the image processor 964 decodes the coded datainput from the external interface 966 or the media drive 968 to generatethe image data. Then, the image processor 964 outputs the generatedimage data to the display unit 965. The image processor 964 may alsooutput the image data input from the signal processor 963 to the displayunit 965 to display the image. The image processor 964 may alsosuperimpose data for display obtained from the OSD 969 on the imageoutput to the display unit 965.

The OSD 969 generates the GUI image such as the menu, the button, andthe cursor, for example, and outputs the generated image to the imageprocessor 964.

The external interface 966 is composed as an USB input/output terminal,for example. The external interface 966 connects the imaging apparatus960 and a printer when printing the image, for example. Also, a drive isconnected to the external interface 966 as needed. The removable mediumsuch as the magnetic disc and the optical disc is mounted on the drive,for example, and the program read from the removable medium may beinstalled on the imaging apparatus 960. Further, the external interface966 may be configured as a network interface connected to the networksuch as a LAN and the Internet. That is, the external interface 966serves as the transmitting means in the imaging apparatus 960.

The recording medium mounted on the media drive 968 may be an arbitraryreadable/writable removable medium such as the magnetic disc, themagnetooptical disc, the optical disc, and the semiconductor memory, forexample. It is also possible that the recording medium is fixedlymounted on the media drive 968 to compose a non-portable storage unitsuch as a built-in hard disc drive or SSD (Solid State Drive), forexample.

The controller 970 includes the processor such as the CPU and the memorysuch as the RAM and the ROM. The memory stores the program executed bythe CPU and the program data. The program stored in the memory is readby the CPU at startup of the imaging apparatus 960 to be executed, forexample. The CPU controls operation of the imaging apparatus 960according to the operation signal input from the user interface 971, forexample, by executing the program.

The user interface 971 is connected to the controller 970. The userinterface 971 includes a button, a switch and the like for the user tooperate the imaging apparatus 960, for example. The user interface 971detects the operation by the user through the components to generate theoperation signal and outputs the generated operation signal to thecontroller 970.

In the imaging apparatus 960 configured in this manner, the imageprocessor 964 has the functions of the image coding apparatus 100 (FIG.1), the image coding apparatus 300 (FIG. 18), the image coding apparatus500 (FIG. 29), the multi-view image coding apparatus 600 (FIG. 34), orthe hierarchical image coding apparatus 620 (FIG. 37) and the functionsof the image decoding apparatus 200 (FIG. 9), the image decodingapparatus 400 (FIG. 21), the multi-view image decoding apparatus 610(FIG. 35), or the hierarchical image decoding apparatus 630 (FIG. 38)according to the above-described embodiments. Therefore, the imageprocessor 964 calculates the quantization value for each sub macroblockand quantizes the orthogonal transform coefficient by using thequantization value for each sub macroblock for the video coded anddecoded by the imaging apparatus 960. In this manner, it is possible toperform the quantization process more suitable for the contents of theimage and generate the coded data so as to inhibit the subjective imagequality of the decoded image from deteriorating. Also, the imageprocessor 964 calculates the quantization value for each sub macroblockby using the quantization parameter such as submb_qp_delta supplied fromthe coding side and performs the inverse quantization. Therefore, it ispossible to perform the inverse quantization process more suitable forthe contents of the image and inhibit the subjective image quality ofthe decoded image from deteriorating.

It goes without saying that the image coding apparatus and the imagedecoding apparatus to which the present technology is applied may beapplied to the apparatus and system other than the above-describedapparatus.

Meanwhile, an example in which the quantization parameter is transmittedfrom the coding side to the decoding side has been described in thisspecification. It is possible that a method of transmitting aquantization matrix parameter is transmitted or recorded as separatedata associated with the coded bit stream instead of being multiplexedwith the coded bit stream. Herein, the term “associate” means that theimage included in the bit stream (or a part of the image such as a sliceand a block) and information corresponding to the image may be linkedwith each other at the time of decoding. That is, the information may betransmitted on a transmission channel other than that of the image (orbit stream). Also, the information may be recorded on the recordingmedium other than that of the image (or bit stream) (or anotherrecording area of the same recording medium). Further, it is possiblethat the information and the image (or bit stream) are associated witheach other in an arbitrary unit such as a plurality of frames, oneframe, or a part of the frame, for example.

Although preferred embodiments of this disclosure have been described indetail with reference to the attached drawings, the technical scope ofthis disclosure is not limited to such examples. It is clear that oneskilled in the art of this disclosure may conceive of variousmodifications and corrections within the scope of the technical idearecited in claims and it is understood that they also naturally belongto the technical scope of this disclosure.

REFERENCE SIGNS LIST

-   100 Image coding apparatus-   105 Quantization unit-   108 Inverse quantization unit-   117 Rate controller-   121 Sub macroblock quantization unit-   122 Sub macroblock inverse quantization unit-   151 Sub macroblock activity buffer-   152 Quantization parameter calculation unit-   153 Quantization processor-   200 Image decoding apparatus-   203 Inverse quantization unit-   221 Sub macroblock inverse quantization unit-   251 Quantization parameter buffer-   252 Orthogonal transform coefficient buffer-   253 Inverse quantization processor

The invention claimed is:
 1. An image processing apparatus, comprising:circuitry configured to set a current quantization parameter for acurrent coding block formed by block partitioning, which recursivelysplits a current largest coding block into smaller coding blocks, and aquantization parameter based on a difference between the currentquantization parameter for the current coding block and a slicequantization parameter for a slice containing the current largest codingblock and the current coding block, quantize transform coefficient datausing the current quantization parameter for the current coding block togenerate quantized data, and code the quantized data to generate a bitstream including the quantization parameter.
 2. The image processingapparatus according to claim 1, wherein the circuitry is configured toset the current quantization parameter for the current coding blockbased on a value indicating a minimum coding block size for which adifference quantization parameter is set.
 3. The image processingapparatus according to claim 2, wherein the difference quantizationparameter is based on a difference between the current quantizationparameter for the current coding block and a previous quantizationparameter for a previous coding block.
 4. The image processing apparatusaccording to claim 3, wherein the previous coding block is a codingblock that is decoded immediately before the current coding block in theorder the coded data is to be decoded.
 5. A method of an imageprocessing apparatus for processing an image, the method comprising:setting, by circuitry of the image processing apparatus, a currentquantization parameter for a current coding block formed by blockpartitioning, which recursively splits a current largest coding blockinto smaller coding blocks, and a quantization parameter based on adifference between the current quantization parameter for the currentcoding block and a slice quantization parameter for a slice containingthe current largest coding block and the current coding block;quantizing, by the circuitry, transform coefficient data using thecurrent quantization parameter for the current coding block to generatequantized data; and coding the quantized data to generate a bit streamincluding the quantization parameter.
 6. The method according to claim5, wherein the step of setting the quantization parameter comprises:setting the current quantization parameter for the current coding blockbased on a value indicating a minimum coding block size for which adifference quantization parameter is set.
 7. The method according toclaim 6, wherein the difference quantization parameter is based on adifference between the current quantization parameter for the currentcoding block and a previous quantization parameter for a previous codingblock.
 8. The image processing apparatus according to claim 7, whereinthe previous coding block is a coding block that is decoded immediatelybefore the current coding block in the order the coded data is to bedecoded.